1.  Good ideas  

People have ideas for features and changes on grist.org all the time, which is awesome.  All of these are evaluated and if possible merged into the overall product plan by the product owners (currently Scott, with some help.)  The product owners are people who are responsible for the overall direction of grist.org.  However, good ideas come from everywhere around Grist — if you have one, draw it out on a piece of paper, write about it on Basecamp or in an email … let people know about it, refine it and sell it (i.e. — let everyone know how it will help us advance toward our goals — growth, engagement, influence and impact.)

Once an idea is sufficiently well defined it becomes a feature request, and is entered into pivotal tracker, our project management software in a place called the icebox.   The product owners regularly review the contents of the icebox and the overall product roadmap, promoting those ideas that we decide need to be done into the product backlog.

The feature backlog and icebox in pivotal tracker. The icebox contains stuff we may do, while the backlog contains stuff we will do.

2.  Design

The backlog is an ordered list.  We do the stuff at the top first, and the stuff at the bottom last.  The order of the backlog may be changed by the product owners frequently depending on changing goals.  When a feature starts moving up the list, it enters a design phase where designers, developers and stakeholders work out all of the various details — how the feature should behave, perform and look.  All of the what-if’s are contemplated, requirements are written down to the extent needed, sometimes there are wireframes, visual designs undertaken …  Design happens before, during and after a feature is implemented.  Sometimes, the design isn’t really formed until a prototype is implemented, which means that this phase can overlap heavily with the next phase which is …

3.  Implementation

We build our software in 2-week chunks called sprints.  At the beginning of each sprint, we commit to the top slice of the product backlog, declare it the “current sprint” and give it an entertaining name. Our commitment to everyone at Grist is that we’ll have this list of features done and live on Grist.org in two weeks.  We also communicate clearly at the beginning and end of the sprint about the status of all of these features, and any adjustments that have been necessary.  Sometimes priorities change mid-sprint, in which case we will usually horse-trade:  we add new tasks if we subtract stuff that we were originally going to do.

During implementation, we have a stand-up each day.  At this short meeting, developers self-organize around the current sprint, committing to new work for the day, and reporting on yesterday’s progress. More concretely, developers commit to tasks in pivotal tracker, and put their names on them.  At the next standup, they’ll report on how things went with that task.

part of a current sprint, showing tasks in various states

After a developer has finished work on a ticket in a special development environment, they tell pivotal tracker that they are done with it by marking it as finished.    Now the feature is ready to be tested.  No one tests their own code, so another developer is then deputized to test the feature.  Usually this means a combination of functional and browser testing, but may mean other things too.  Before testing however, the feature must be delivered.   Features are delivered when they are committed to our code repository.

A quick list of pivotal tracker states:

- tasks that have not started have a gray background
- tasks that have been started are yellow, and will indicate their next possible state via a button. (you can also open the ticket and see the state)
- accepted tasks have a green background
- deployed tasks have a green background and are tagged with the word “deployed”

There is also a physical manifestation of the current sprint in the office. It’s on the wall near the tech area — you’ve probably noticed our raft of sticky notes etc. This board represents the current sprint and mirrors pivotal tracker.

3a.  Nerd Interlude

(read this if you want but feel free to skip.)

We employ the following general philosophy regarding versioning/repos:  We maintain a clean daily svn trunk.  That means we don’t commit code that is broken, and we isolate large or difficult-to-implement features in branches, and then merge them back into the trunk when testing is complete, after an integration test.  Furthermore, the trunk must be fully tested and clean by 11pm each night, at which point it is tagged, and eligible for deployment.  (end interlude)

4.  Testing 

Once a feature has been delivered, it is quickly tested.  Small features are tested the same day.  Larger ones might take a bit longer, as there are sometimes bugs.  Once a feature passes its test and is integrated into the main applicaiton, it is declared accepted.  Accepted features are eligible for deployment.  

5.  Deployment

Deployment is the process of pushing a completed feature to Grist’s production environment, usually the one on the wordpress.com cloud.  In order to deploy a feature, we transfer it from our codebase to the codebase on wordpress.com, and wait a short time for wp.com engineers to review and deploy it.  As a result of clean, tested nightly tags of the trunk, we can deploy grist.org (i.e. — all accepted features) at least once each day.  The pause between us doing a deployment and it appearing on grist.org can be hours long at times.  It is often shorter.  Once a feature is deployed, it’s visible to the world on grist.org.

6.  Iteration

Thought we were done?  But no!  Now comes the part where we gather data on deployed features, continue design (that never ends) and come up with new ideas about how to improve what we see happening with Grist in the real world.

In summary, you can think of the progression like this:

idea -> definition -> icebox (via product owners) -> backlog (via product owners) -> current sprint (via tech team) -> started (when a developer begins work) -> finished (when a developer finishes work) -> delivered (when it enters our codebase) -> accepted (when it is tested) -> deployed (when it appears on grist.org) -> iteration -> …

Jargon:

Here’s a list of definitions you may find useful:

Basecamp:  a platform we use to discuss design, features and other stuff.  Created by a religious organization known as 37Signals.
product owner:  someone who is ultimately responsible for the product — it’s their job to make sure grist.org meets its goals and is great
feature request:  a clear idea for a new feature — it’s good to talk about and refine these a lot
Pivotal Tracker:  some software we use to organize our work, and track lots of features and bugs
icebox:  a collection of features we may implement
backlog:  a collection of features we will implement
design phase: a phase before, during and after an idea becomes a real thing.  It’s where the plans, visual aspects, inner workings and other details of a feature are hashed out
designer: someone who is smart and designs stuff.  Colloquially, a visual designer
developer: an awesome, incalculably attractive person who works near the middle of the Grist offices.  Also, someone who writes code, designs and implements web applications
stakeholder:  a person who has a business interest in the product.  This interest might be “raising money”, “publishing content” or “engaging community members”
sprint:  a two week period in which we commit to doing a bunch of work
stand-up: a meeting where everyone stands up (so that the meeting is short.) We use them to self-organize during sprints
finished: the state a feature is in when it requires no further programming
delivered: a finished feature that has been entered into Grist’s codebase
committed: a synonym for delivered
accepted: the state a feature is in when it has passed all tests
deployed: a feature that is live on grist.org, or about to be
repository: an iron fortress that contains the current state of our codebase, and all previous states
trunk: the version stream of the product from which we deploy
branch: side-streams of the product that we use for development purposes
iterate: to make repeated use of the development sequence defined above.  Generally produces improvement (which is beyond definition, and in the eye of the beholder.)

Today at Grist we celebrate the end of our transition from our co-located infrastructure to the cloud.  This change, which included moving our CMS and primary hosting, migrating our donation and membership applications and changing our email broadcasting technology, has seemed like a long process at times, but has resulted in a Grist that is infinitely more flexible and ready for growth than we were at this time last year (and given how we’re growing, that’s a good thing.)

If you’re curious about where we’re hosted now instead of on these boxes, we’re hosted on the WordPress.com cloud through Automattic’s VIP .com service and on Amazon AWS.

The boxes pictured here powered grist.org and related stuff for over 3.5 years.  Most were continuously running (disks spinning) throughout that whole time.  Personally, I hope they enjoy the rest.

What occurred to us at some point is that this sort of software has a very general use case for almost any news or advocacy organization covering a quickly changing topic, and so we decided to build not only a realtime news application, but the framework that would support arbitrary applications of this kind.  We’re calling that project CanopyEngine, and you can read more about it in this application to the Knight News Challenge.

From a design and technical point of view, we’re very excited about this first step — certainly because we think the design is cool, but also because of how we’ve decided to approach this sort of evolution on Grist.

There are fewer activities more fraught than the undertaking of a major web redesign.  These sorts of projects generally take forever, follow circuitous and jerky routes through a forrest of requirements and edicts, and often end up somewhere short of where you’d hoped they would.  Plans are laid out in a linear, (waterfall) fashion, and often extend stiffly into the future horizon.  Likewise, scope tends to balloon, meaning that redesigns rarely confine themselves to a particular product or problem, but spread across the available landscape via an often malignant web of dependencies and good intentions.  Change becomes ever more expensive.  Risk increases too, as does cost.  Things look something like this:

Like many small web teams, Grist has been using agile methodologies to help us manage our web development activities for some years now.  We’re now extending this philosophy into the design and product development process too.  Nowadays, our design activity adheres to some general principles, but otherwise concerns itself with a series of smaller changes or products that are deployed much more quickly and then iterated upon.  This allows for rapid change in priorities, lets us adapt to unforeseen problems, and keeps us constantly thinking about only the decisions we need to consider.  Visually:

This approach is not without its pitfalls … for example, maintaining design unity and consistent state for the whole site is more difficult when designing in this way.  However, what we’re aiming for is a pleasant, continuous, rolling evolution that is actually less jarring that the traditional massive-end-reveal that often characterizes redesigns.

As long as we have a set of principles to which we adhere, we think that this approach will prove effective … only time and (more importantly) our readers will tell.

Even though  you may not see any overt changes, today was relaunch day at Grist.  We turned off our old site, and turned on a new, WordPress-based version.

What we’ve done is known as a “port” — that is, we’ve taken the Grist UX and content, jacked it up, and replaced the old foundations with new, better WordPress ones.  This required many many weeks of effort, some valuable expert advice and lots and lots of coffee.  It also meant that @natebot and I have had to learn the WordPress API rather quickly … luckily, this has been fun.

For more of a public intro to what we’ve done, please read this post on Grist by Scott Rosenberg, our Executive Editor.  Notice a bug with the content or site? Please report it here.

Grist is adopting WordPress for two main reasons:  WordPress has become the world’s foremost CMS, and is the locus for a whole lot of journalism innovation and experimentation.  We want to be part of that community, and more able to take advantage of the latest online news technologies.  Our other goal has more to do with who we are.  We are small — really small — and as such want to focus our technical energies on news, design and innovation, rather than hosting, uptime and infrastructure.  Our new hosting arrangement with WordPress.com VIP allows us to do that, and lets us join the ranks of other large media sites like Time, Chicago Tribune, and these guys from down the street.

So while today’s change might be almost invisible to most, we can tell you that there will be more exciting, and more rapid changes on the way, so stay tuned!

That’s not to say there existed no news sites/apps based on the Twitter API before the last six months. However, whereas there is a lineage of Twitter mashups out there, these tend to be fairly rudimentary in nature in the sense that they don’t implement much (if any) intelligence on top of twitter streams, but simply aggregate and display them in clever ways. And while the big search engines and Twitter’s anointed resellers have been consuming the Twitter firehouse and implementing all sorts of intelligence/NLP/indexing over it for some time now, these sorts of activities (even on the level of link counting and indexing) were beyond the powers of news developers.

That has now changed.

So if we have a new genre on our hands, we have to name it right? I’m going to call these sorts of apps news smashups. (ok so it’s kind of a dumb name, and I know it can also mean “bad automobile accident” … but humor me.) Definition:

A news smashup is a news-oriented web application that:

• consumes streams of content primarily from social sources (Twitter) in near real time.
• applies intelligence/indexing/filtering to these sources, also in near real time.
• implements a web experience that to the end user appears to be edited, curated or guided by a subset of the social web.

Drawn completely from my own biases and experiences, here are a few illustrative (and in some cases mysterious) examples of this type of thing:

Muckrack!  What started as everyone’s favorite Twitter mashup for collecting, categorizing and streaming tweets from a big pool of journalists is now replete with bells and whistles, including new pro features like search and alerts.  This is a great example of a curation/aggregation Twitter application focused on a particular vertical. It’s also an example of how a mashup can become a … smashup!

From the somewhat mysterious department, there’s Percolate: a Twitter aggregator powered by a set of relevance algorithms and feedback mechanisms … but with much higher aspirations.  From the horse’s mouth:  Percolate works by hooking up to streams of content … and filtering down to the most interesting stuff for you . We then present that content back to you for you to react to, which is as easy as hitting an “awesome” button.

Sociative develops technology to use “social signals” (largely from Twitter) to identify content signal amid a huge amount of content noise.  Applications of their technology so far include several socially-curated sites, most recently a socially driven representation of news from the occupy movement called Occupy Live.

Andrew Phelps, prototypical journo-hacker, built Fuego, The Nieman Lab’s “heat seeking twitter bot.” Fuego collects the most popular content from a curated set of media experts in realtime, and exposes them as a simple list of trending material based on link counting and other weighting.

I’m sure that this is an entirely incomplete list. I’d love to hear what others think about defining this as a genre. Does it make sense to do so? If so, what are some other examples of this sort of thing in the news apps space?

Last week we were lucky enough to play host to two members of the WordPress core team last week here in our offices in Seattle.  If you ever get the chance to work with/learn from Otto (@otto42) and Nacin (@nacin) … well just do it!  Grist would not be what it is without the contributions of many people, but it many ways it’s contributions of time and energy by experts like Otto and Nacin who make doing what we do possible.  So thanks guys.

More soon on what exactly these guys were doing with us last week, all we can say is that we’re excited!

IronCache is an extension for the ExpressionEngine CMS that boosts performance for high traffic sites by using memcached, the world’s most popular general-purpose memory caching system.  We’ve been using it here at Grist for about 18 months on our EE 1.x-based site.  We’ve ported it to EE2, and are now releasing it.

More information, downloads, documentation, etc. is here!

Here’s what happened:

Team Halliburton (Nathan, Wade and Dave) worked up an interactive timeline of last year’s oil spill, featuring cool “sinking rig” graphic and movable map background.  The timeline data was sourced from data.gov.

Team Exxon (Libby, Cindy, Darby, Matt) produced a post about last week’s tornado activity based on historical tornado data.  They used Excel and plain ‘ol photoshop to produce the following trend chart:

Then, they used openheatmap to make the following animated map/timeline of tornado severity (which it should be said, lacks a certain amount of context but is cool anyway):

http://www.openheatmap.com/embed.html?map=SlenderestUnderskirtsDizzying

Team Monsanto (Darcy, Hanna, Lisa, Dan) explored the relationship between walk score and political affiliation, and discovered that the only  (ONLY!) walkable Republican city is ……….. Oklahoma City.  They also used wordle  to analyze recent Grist content, resulting in this:

Grist Keyword Content in March

This was a pretty successful day from our point of view.  At the very least, it has given us some valuable insight into what it takes to do this kind of work, and how it might happen here more often.

Coming soon and throughout the year,  look for improvements in the facebook sharing mechanism, and further streamlining on these tools.

More generally, we’ll be working on the entire article-reading experience to make it cleaner, more social and more immediate.