The ProPublica Nerd Blog

Resources for Investigating Tax-Exempt Organizations

2013-05-09T15:28:57-05:00

In addition to our Nonprofit Explorer interactive database, here are some resources for researching charities and other tax-exempt organizations.

Getting Started

The Reynolds Center for Business Journalism has an “Investigating Nonprofits Tipsheet” as well as self-guided training on investigating nonprofits (as well as other private companies)
The Foundation Center’s 990 Finder has information on U.S. charitable foundations, including financial reports.
Guidestar Premium has sophisticated tools that let you compare nonprofits by financial health, number of full-time employees, board chairs, etc.
The National Association of State Charity Officials links to state offices that regulate charitable organizations and charitable solicitations.
You can request 990s or applications for exemptions directly from the Internal Revenue Service. The IRS website also allows for searching of political organization disclosures and has 990-n postcard searching.
Some states, such as North Carolina require nonprofits to file their annual audits online.

Diving Deeper

The Indiana University archives retain some historical non-profit filings, and you can submit search requests.
NCC of New York and Ford Foundation have a very in-depth guide on how to read a 990 to analyze their net assets, look for self-dealing, find out if any officers have left, etc.
The law that covers tax-exempt organizations is 26 USC § 501 Exemption From Tax on Corporations, Certain Trusts, etc.
The IRS publishes many useful guides for tax-exempt filers, which are helpful for those trying to understand how nonprofits work, including: Instructions for Form 990 Return of Organization Exempt From Income Tax and Publication 557: Tax-Exempt Status for Your Organization
The Chronicle of Philanthropy publishes two particularly useful resources: How America Gives, a database that looks at giving patterns in every city, state and neighborhood in the U.S., and America's Top Donors, a database of gifts of $1 million or more to charities since 2003.

A Super-Simple Tool to Search Instagram by Time and Location

2013-05-03T15:04:44-05:00

This week, Justin Elliott wrote about new House Financial Services Committee chairman Rep. Jeb Hensarling (R-Texas) attending a weekend getaway with banking industry officials.

One of the ways he found out who was at the getaway was by using the Instagram photo sharing service, which turned up a snowy snapshot taken by Len Wolfson, a lobbyist for the Mortgage Bankers Association (which had contributed to Hensarling's PAC). Wolfson has since set his account to private.

The Instagram site has no search function, so finding shots like this can take a lot of digging. However, Instagram has an API with a "Media Search" endpoint that returns data both by timeframe and distance from a certain latitude and longitude -- a perfect way to see who's at a certain place at a certain time.

We wrote a simple Sinatra app that uses this endpoint which we're calling QIS (Quick Instagram Search). The API is pretty limited, so our tool is too. There's no way to search for text or hashtags (tags have their own endpoint which doesn't allow geolocation), there's no pagination of results, and results only go back a few months. We're using Google's Geocoder to take any place, landmark or address string and turn it into a latitude and longitude for Instagram. We're open-sourcing QIS today.

Here's an example of a search for yesterday between Noon and 7 p.m. at the New Orleans Fair Grounds, where the New Orleans Jazz and Heritage Festival is currently going on:

Just playing around with QIS for a few minutes, we found a shot of the finish line at the Boston Marathon, 8 minutes before the first bomb exploded, and a shot of Newt Gingrich outside the White House Correspondents' Dinner.

To bootstrap QIS, see the README file in the repository.

Tracking State Bills Takes a Village: The Legislature Tracker Gets Launched in N.Y.

2013-05-03T14:00:30-05:00

I'm an interactive news developer at MinnPost in Minneapolis. I've been working in the Nerd Cube at ProPublica this week as a P5 Resident.

I spent the week working with other developers enhancing MinnPost’s Legislature Tracker so that any newsroom can easily use it to report on the work of their state legislature.

The Legislature Tracker is an application that provides a curation layer on top of the vast amount of legislative data that is produced each session in all the 50 states around the country. It enables newsrooms, organizations or individuals to easily identify and keep track of bills that they or their audiences will find important.

The goal of the week was to make the tool work for states other than Minnesota. That required some coding help but also needed expertise in a statehouse other than my own. So in addition to ProPublica's help we were really happy to turn to the energetic team at The New York World.

After a week-long sprint, we accomplished two major things: We've made the Tracker easily customizable to work with any legislature; and the New York World is launching a New York Legislation Tracker which highlights what their editors have decided are the important bills that are being debated in Albany.

The MinnPost data team, Tom Nehil, Kaeti Hinck, and I, worked hard on developing the Legislature Tracker at the beginning of the year, and we were very excited to launch it last month. We saw early on that this sort of application could be very useful for other states and other groups, and we made many decisions to ensure that the application could be easily re-used and deployed. But due to deadlines and limited resources, we were not quite able to fully reach that goal. Fortunately I had the honor to come be a part of the P5 program here at Propublica and knew right away it would be a great opportunity to reach the goal of finishing what we started with Legislature Tracker.

At the beginning of the week Scott Klein, Jeremy Merrill, and I were talking about what we could accomplish this week, and we realized that the easiest way to make sure we were making the Legislature Tracker widely applicable was to make one for another state. This would make it obvious if our assumptions were correct and if our code worked like we thought it did. Scott immediately thought of The New York World and called editor Alyssa Katz, who jumped at the chance. The next morning two terrific NY World staffers -- Beth Morrissey and Michael Sullivan -- were hacking with us in the nerd cube. Beth is a reporting fellow with knowledge about the New York State Legislature and had contacts to dig in deep on the subtleties of the state legislative process, as well as to identify the bills worth tracking. Mike, who is the New York World’s tech-savvy deputy editor, hopped on to help code on the Tracker and deploy it for New York World.

With this hard-working ad-hoc team, we sprinted all week to add features, fix bugs, identify bills, code in legislative process, and ultimately launch the Legislature Tracker on the New York World site.

I came into the P5 program very excited and ready to make the Tracker project much more useful to everyone, but my expectations have been blown away and am very proud of what we were able to accomplish this week. Many thanks to MinnPost, Propublica, and New York World for the all their help and resources. If you want to launch the Legislature Tracker in your state, shoot me an email.

The Legislature Tracker is all open source. Examples and documentation are over on Github. There's still plenty to do to make it better, so any help is much appreciated.

Latest P5 Resident Working on Open Source Legislation Tracker

2013-04-29T13:29:59-05:00

Our P5 Resident this month is Alan Palazzolo from MinnPost. He’s the fifth P5 Resident.

Alan is an interactive news developer at MinnPost, where he builds open source news applications and data visualizations. His projects have included a live 2012 Minnesota election results dashboard and an animated map of a day in the life of Twin Cities “Nice Ride” bike share. Before joining MinnPost, he was part of the inaugural 2011 Fellow class at Code for America, working in Seattle, Wash. While at Code for America he helped build Change by Us.

While he's here in the nerd cube, he’ll be refactoring his code for MinnPost’s Minnesota Legislative Bill Tracker so it can work for any state. It’s already available on GitHub but we’re hoping by the end of the week to release a new version that will let anybody easily alter it to work with their state's legislative data. The tracker, which uses OpenStates data lets people focus on the bills that really matter to a publication or community.

We’re always on the lookout for new P5 Residents. Here are some details on the program and how to apply.

ProPublica Projects—and P5 Projects—Finalists for Data Journalism Awards

2013-04-29T13:25:38-05:00

Over the weekend, the Global Editors Network announced finalists for its 2013 Data Journalism Awards.

On the shortlist are three projects from ProPublica:

Maps that retell the “Great Migration” that took place in the mid-20th century, which saw African Americans migrate from the rural south to northern cities only to find themselves facing housing segregation in their new hometowns. The project, by Jeff Larson and Nikole Hannah-Jones, accompanied Hannah-Jones’s investigative project about the historical failure of the U.S. Department of Housing and Urban Development to take steps to end housing segregation.

Message Machine, a project to reverse engineer political messaging in the 2012 campaign. News app developer Jeff Larson developed a system that processed, in real-time, political emails sent in by readers, and tried to uncover the factors behind the sometimes subtle differences between the “micro-targeted” messages. With the help of more than 600 readers, we were able to uncover the versions of each email, and the demographic factors the campaigns used to tailor their messages while the campaign was still going on. To our knowledge this was the first time these techniques were used in a news application.

The ProPublica Nerd Blog, where we publish all of the methodologies, recipes and the secrets behind our news application development. It’s always been an important component of ProPublica’s mission to share our work. Our journalism is available to republish under a Creative Commons license, and has been so since we launched in 2008. The News Apps desk at ProPublica makes lots of our code available via GitHub. We also, after each big project launches, explain our techniques in nerdy detail via long technical posts. You can find all of this in the nerd blog.

Also on the shortlist is an interactive project in the Washington Post by Sisi Wei, who now works at ProPublica, called “Exit Polls 2012: How the Vote Has Shifted.”

We’re very pleased to be nominated alongside the amazing work of so many of our colleagues around the world. We especially wanted to point out two projects done by Residents from our P5 program who were also nominated: La Nación (Argentina) is nominated for their remarkable work on Senate expenses, and Berliner Morgenpost is nominated for their “Flugrouten-Radar” project, which got its start as Julius Troeger’s P5 project while he sat in the ProPublica nerd cube here in New York.

The winners of the 2nd annual Data Journalism Awards will be announced in Paris on June 19. A full list of nominees is available on the GEN website.

HeartSaver: Experimenting with News Games to Tell a Story

2013-04-23T10:51:25-05:00

This past weekend, a team from ProPublica competed in the GEN Editors’ Lab New York hack day, with the theme “Newsgaming.” We learned how to use game mechanics to create an interactive experience that went beyond badges and reputation systems to explore a complex accountability story in fun and engaging way. Seriously, our game is really fun.

Here’s what we learned (and produced!) in two days.

The game, HeartSaver, aims to help players explore access to good emergency care in New York City, where nearly half a million people suffer from heart disease. We wondered: does everyone have access to high-quality care? What happens if I have a heart attack at work? At home? We invited players to explore this question by challenging them a mission: Save as many heart attack victims as possible.

A player’s score depends on how quickly they can get heart attack victims to the best available hospital. For victims, survival often hinges on how quickly they can get to the emergency room, with odds decreasing 7-10 percent every minute before defibrillation, according to the American Heart Association. And on arrival, quality of care plays an important part. We turned these factors into variables in the game. Players can explore how access to good emergency care works. The faster a player routes victims to quality care, the more lives they save. The more lives they save, the higher their score. How many lives do you think you could save? Play the game and find out.

News Games Tell a Story

While playing the game, players are telling themselves the story we want them to tell. It becomes clear very quickly which areas have fewer hospitals with emergency departments. When an icon representing a victim lands in Woodhaven, Cypress Hills or Whitestone, Queens, the closest hospital suddenly seems very far away. Having players experience the anxiety of seeing a victim with no hospitals nearby gives them an intuitive and memorable understanding of how the lack of emergency care affects neighborhoods -- much more so than they’d get from an article or an interactive map.

As the difficulty of HeartSaver increases, players are faced with two important and emotional decisions.

First, if a victim appears close to a hospital with a below average heart-care rating, does the player risk a longer transportation time to bring the victim to a better hospital? The precious seconds players take to decide affect the victim’s survival rate.

Second, as the number of victims increase, it becomes difficult to prioritize them. Should a player triage victims based on which ones needed help first, or focus on helping those already close to a hospital?

Forcing players to make these decisions lets them actually experience the complexity and difficult decisions in the real system.

Using Real-ish Data in a Game

HeartSaver relies on three key data points: The time elapsed since each victim’s 911 call (how long it takes the player to drop the victim icon onto a hospital), the time it would take to transport the victim from their location to that hospital in the real world, and that hospital’s relative quality of care.

In the game, a minute is compressed into a second. For every second that passes as the victim awaits help, survival odds decrease 10 percent. For every second of game time spent driving, survival odds decrease by .58 percent. Since we couldn’t find solid data connecting minutes of ambulance travel time with survival odds, the game uses the lowest rate we could find. We calculate the survival rate once the victim arrives at the hospital using that hospital’s mortality rate. If you’re a researcher and can help us answer this question more rigorously, let us know.

We calculate transport time from a victim’s location to the hospital the player drops them onto by passing addresses and coordinates to the Google Directions API, which returns the estimated driving time.

We also use Medicare’s risk-adjusted 30 day heart attack mortality rates to factor hospital performance. In the game, a victim’s post-travel survival chances are multiplied by Medicare’s survival rate for heart attack victims at that hospital. Finally, there’s an element of randomness applied after all of these factors are combined.

Here’s the full formula for how we calculate survival chances:

Some important caveats: We don’t know if a hospital's mortality rate factors out things like response time and travel distance. Also, in the real world, there isn't just one ambulance dispatcher for the city, and you'll almost definitely end up at the nearest hospital to your current location. Finally, the American Heart Association statisitc we use refers to sudden cardiac arrests, which is different from heart attacks, though a heart attack can also lead to sudden cardiac arrest. Read more about that here. This is a two-day hackathon project meant to explore how news games work, and not the results of months of rigorous research, so take things with a grain of salt (actually, you should probably go easy on the salt).

In order to make sure that every “patient” landed in one of the New York City boroughs, and not, say, in the water or New Jersey, we used geographic shapefiles from the NYC Department of City Planning.

News Games vs. Interactive Graphics

We could have made a great interactive graphic on heart disease and emergency care in New York, plotting hospital locations and quality of care ratings relative to reader locations. But that’s not a game.

On the first day of the hackathon, Columbia Journalism School professor Susan McGregor spoke to the teams about game design, and three of the criteria really stood out to us.

Games have an objective: Some games like Minecraft can pull off being a sandbox, which is basically an open-ended game where players can do anything they want, without any goals. But for most games, the objective is what makes it fun, because people enjoy the feeling that comes with overcoming challenges.
Result of the game is unknown: Players don’t know ahead of time how the game will end, and more importantly, players can take actions that impact the outcome. The game isn’t passive, instead it actively engages the player in a series of decisions and actions toward a final goal.
Creating the feeling of “together, apart”: A great game allows players to feel a sense of community, even though they are sitting with their own devices, looking at their own screen. Think of the success of massive multiplayer online games (or MMOs), such as the World of Warcraft. Our goals for HeartSaver were much more scaled back, but we wanted to create a “together-apart” feeling by giving players statistics on how their score compared with others who played.

If you’re interested in learning more about news games, Sisi also gave a lightning talk on how to make them at the NICAR 2013 conference.

Next Steps

Because this was a two day hackathon project, we were pretty constrained as to what we were able to accomplish. If we had more time, we would have been much more rigorous with the data, and added more opportunities for players to see access to high-quality care near them.

Also, we would’ve allowed players to input their own addresses for work and home (or link up their Foursquare account to detect popular locations) to see how far their nearest hospital was. Then, we would’ve encouraged them to share and check hospital distances for family and friends. Once we’ve got readers’ attention with the game, we would have loved to show them an immersive interactive graphic to show how quality of care in the city affects them personally.

We want to thank the Global Editors’ Network for such a great event, as you can tell, we’re pretty excited about what news games can do. If you haven’t already, check out HeartSaver, which is currently still a prototype, and make sure to play the all the other games that came out of the hackathon!

Meet the New Blog (Same as the Old Blog)

2013-04-16T10:07:40-05:00

Today we debut a ProPublica Nerd Blog redesign, which includes a new look for our main landing page as well as our article pages. We wanted to create both a better reading experience, as well as to help you find what you’re looking for more easily. Here are three new features we want to highlight:

First, we created a new sidebar to help you quickly locate our tools and guides, including ProPublica's latest News Apps and Data Guides. It’s powered by the Github API, so anything we open source will instantly be added there. We've also added a section called "Explore Our Work" that gives you another way to navigate through our stuff.

Second, we're cross-posting announcements all of our news apps and graphics here. This makes the Nerd Blog a one-stop shop for all of our projects and the technical posts we write about them. Look for the gold boxes for new apps.

Third, we added simple keyboard shortcuts to the landing page to help you navigate more quickly. You can press "j" to skip to the next blog post, and "k" to go back. Vim fans should be delighted.

We hope you like the new look, as well as our new "data bass" icon, drawn by Jason Das, and we welcome suggestions on how to make the nerd blog even better in the comment section -- now powered by Disqus -- below.

Heart of Nerd Darkness: Why Updating Dollars for Docs Was So Difficult

2013-03-25T11:24:46-05:00

Last week we published a big update to Dollars for Docs, our interactive news application of payments made to U.S. health-care providers by 15 pharmaceutical companies. Compared to when we launched the project in 2010, the amount of data we’re collecting has grown enormously: The list of payments increased from around 750,000 to almost 2 million, and the grand total of the payments grew from around $750 million to just under $2 billion.

Compiling the data for it has been an enormous project right from the beginning. After we published the first version, the original developer on the project, Dan Nguyen, compiled all of the things he had to learn into a guide to scraping data. This year’s update took more than eight months of full-time work by me, working with other news-app developers, and at times with our CAR team, a researcher, two editors and two health-care reporters. It was a massive effort and presented huge technical and journalistic challenges.

After we launched, my editor pulled me aside and asked what was so hard about Dollars for Docs. What follows is my answer.

PDFs Considered Harmful

To build the database that powers Dollars for Docs, we had to convert 65 gigantic disclosure reports from some other format into raw structured data. Each company releases its data in a slightly different way, but broadly speaking there are two ways they present the data: A few build websites to publish the data, and many more disclose their payments in humongous Adobe Acrobat PDF files.

The trouble is, PDF was not designed as a data format. It was designed as an "electronic paper" format. That is, "something whose contents (text and 2D images) would look the same on any computer at any time," Adobe Senior Product Marketing Manager Ali Hanyaloglu told me.

PDFs are a format engineered to present elements in perfect fidelity to their creator's intentions. In their most basic form, PDFs don’t know what tabular data is. They don’t even know what words are.

Here’s how a PDF works, deep down: It positions text by placing each character at minutely precise coordinates in relation to the bottom-left corner of the page. It does something similar for other elements like images. A PDF knows about shapes, characters and their precise positions on the page. Even if a PDF looks like a spreadsheet -- in fact, even when it’s made using Microsoft Excel -- the PDF format doesn’t retain any sense of the “cells” that once contained the data.

Adobe, which invented the PDF in 1993, acknowledges its shortcomings when it comes to data. “For the person who wants raw data, PDF isn’t the right choice," wrote Adobe Senior Principal Scientist Jim King in a blogpost. In 2000, Adobe made an updated version of the PDF format with an easy way to attach files so that original data could be published alongside an attractive presentation.

Incidentally, Adobe made doing this really easy. We made a screencast showing how. According to Hanyaloglu, it can be done using Adobe Acrobat, Microsoft Office or even with the free Adobe Reader. I’m not quite sure why I don’t see PDFs with attached data in the wild more often. In any event, none of the 50 PDFs we used to compile Dollars for Docs database attached the raw data.

For each PDF, we had to reconstruct the raw data, and it’s not an easy thing to do. We don’t know of any off-the-shelf software that did precisely what we needed to do, so we wrote our own. We named it "Farrago." The first version was built by David Frackman, a New York-based developer who recognized this as a problem that could be addressed with a computer science discipline called computer vision, which is used for things like facial recognition and automatic license-plate readers.

Farrago reconstructs the data from a PDF by considering each character one by one to determine which column it belongs in. It uses a computer vision technique called a Hough Transform to detect the vertical black lines in the background of the PDF that visually separate columns in the table.

But not all PDFs have borders around each table cell. For these PDFs, we had to visually guess the locations of these dividing lines and adjust the code again and again until it came out right. Unfortunately, there's no easy way to check if tens of thousands of lines were all copied correctly. Many times we had to reprocess a file because one person's long name was inadvertently split, with most of their name in the name column and the last few letters in the next column. For example, an incorrect column dividing line location for a payment to a doctor with the last name "Ingemar" in Salt Lake City might end up listing "INGE" as a last name and the city as "MARSALT LAKE CITY.”

Even when Farrago has an accurate model for which columns everything belongs in, things aren't easy. As I mentioned above, each letter is positioned on the page independently from other letters in the documents. That's more inconvenient than it may sound: While letters like "a" or "c" have about the same location on the y-axis if they are on the same line, letters like “p” or "g" with descenders are a tiny bit closer to the bottom of the page than letters without descenders, like h and m. Punctuation marks like commas and periods also behave unexpectedly, for the same reason. Farrago "rolled up" the letters within an arbitrary number of pixels of the baseline into a single line of text.

Here’s another mystery we found. Some PDFs we processed seem to contain duplicate lines of text that weren’t there if you just looked at the PDF. We had no idea why. The PDF specification, which is the official recipe for how to write a program to display, print and create PDFs, made no mention of this phenomenon.

A paper by Tamir Hassan, a computer scientist whose research focuses on table recognition and information extraction, may have solved the mystery. “Characters (or complete strings),” he wrote, “are sometimes overprinted with a slight offset to simulate boldface type. As long as these instructions follow another, they are automatically detected and represented by a single text fragment with the boldface flag set to true.”

Finally, because PDFs don’t know about words, they often don’t include a space character separating them. Instead, PDFs simply locate each group of letters a few pixels to the right of the one that precedes it. A reader can’t tell the difference, but to a computer the difference is a big pain. We used some PDF-reading software libraries that made sometimes imperfect guesses about where to put spaces in.

Even after we had Farrago doing the heavy lifting, it often took hours to precisely tune how it processed a given PDF, and it sometimes took days to discover subtle problems and write a fix or workaround.

Breaking the Web

If they follow best practices, websites are easy to turn into structured data. But pharmaceutical companies' websites don't always follow best practices. Some clumsily reinvent the basic mechanics of the Internet and make “scraping” the data out of them exceedingly difficult.

One company's disclosure site spanned 4,000 web pages that all shared a single URL. That is, there was no way to get to page 3,000 without viewing page one, then page two, then all the intervening pages through page 2,999. This wouldn't be so bad, except the company's web servers often got overwhelmed and returned a broken page, forcing us to start over.

After I re-engineered the code to work around the site's limitations with my colleague Jeff Larson's help, the server would still occasionally get confused, and in unpredictable ways. If we requested, for instance, pages three and four, the server might return pages labeled "3" and "4" but both containing the data intended to be on page four.

Another company's disclosures seemed, at first, to all be on a single 29.6 megabyte web page. It took more than two and a half minutes to download and render in Google Chrome -- on a very fast Internet connection. But it turns out this behemoth of a web page did not even include all of the disclosure data. It included only the total amount paid to each payee. The details about each payment were on a fragment of HTML that appeared when a user clicked a link. We had to download all 81,335 of these separately and merge them with the list of payees so that we could report which portion of each payment was for meals, consulting, speaking, travel, etc.

It Gets Worse

Even when all the data was downloaded, we faced a row of hurdles.

There is no standard way for companies to define each kind of payment -- at least, not yet. The 15 companies describe categories of payments in different ways and some even define categories differently between their own reports. For instance, while many companies disclose the value of meals given to doctors, Merck combines meals, travel expenses and royalties, so there is no way to break out the value of meals paid for by Merck to compare it to similar expenses disclosed by other companies. In all, we tracked 155 distinct category definitions across all of the companies in Dollars for Docs. The specific definition created by each company for its payment categories is available as a “tool-tip” next to each payment on Dollars for Docs payment pages.

Another hurdle was dealing with the small number of companies that disclosed payments as ranges rather than precise dollar amounts. For instance, Allergan reported paying between $1 and $1,000 to a doctor named Aaron Jacob Friedman in Lafayette, La., for "business meals" in 2011. There is, of course, no way of knowing if that was lunch at McDonald’s for $3 or dinner at Le Cirque for $999. Multiplied by the 95,582 payments in Allergan’s disclosures, this was a big problem. In the end we reported the ranges as we found them and excluded all the ranged payments from our aggregate numbers, like state and company totals.

Because any doctor's name is likely to be reported differently by each drug company, we had to be careful about how we reported what was paid to each doctor. Some companies list middle initials and others don’t. Some checks appear to have been written out personally to a doctor, others to the doctor's practice. Some doctors' addresses are listed in multiple cities. Manually researching millions of records to determine which slightly different names are really the same person and which doctors share a name but are different people would be an impossible task. We chose not even to attempt it and instead followed the example set by the Center for Responsive Politics' OpenSecrets site. We show names as we find them and depend on users to know which records in a search result page refer to their doctor.

For doctors who stood out -- like the 22 doctors who earned more than half a million dollars -- our reporters confirmed that the payments were all to the same person.

All of this made comparing the data and drawing conclusions very complicated. For instance, because we couldn’t with high confidence group all of a doctor’s payments together, we couldn’t report the highest paid doctor in each state. Luckily, some of the news organizations who’ve done stories based on Dollars for Docs data did the necessary legwork and do name the most highly paid docs. We also could not compare some companies payments to those made in a previous year. For instance, Merck's 2012 payments can’t be compared to its 2011 payments because they changed what types of payments they disclosed. Nor could we compare companies payments against each other, as each company discloses different types of payments.

Making Sure We Get It Right

After the data was collected, we worked very hard to make sure that every data point in our database was accurately reconstructed from its source. The techniques we used are detailed in my colleague Jennifer LaFleur's data bulletproofing guide.

During more than a dozen rounds of spot checking, my fellow ProPublicans used statistical sampling to make random selections of hundreds of records at a time and manually compared them to the record in the original PDF or website. We also manually recomputed every sum and average -- for aggregates like companies and states -- using Microsoft Access and compared them to the version calculated by our Ruby code.

We also used a SQL “GROUP BY” query to get all of the unique state abbreviations in the data set. This helped us locate errors like "M I" (with a space) instead of "MI" (without the space) for Michigan. We then sorted every column to make sure that it didn't include numbers in alphabetical columns, like city names, or letters in the amount column.

Another important component of our data-bulletproofing system was keeping our code and data well-organized. Each step of the data-cleaning process had its own script. For instance, one script used Farrago to scrape a PDF and filtered out header rows. For each step, we saved each of these scripts' output. This made it easy to track down errors. If, during one of our spot checks, we found a line that was missing in our final data that was present in the source data, we could look at each step of the data-cleaning process for that document to pinpoint which script had introduced the problem. After we had fixed the problem, the whole process could be re-run with just one command.

Help is On the Way

The Physician Payment Sunshine Act may resolve many of the problems we had. A provision of the Affordable Care Act, the PPSA will require pharmaceutical companies and medical device manufacturers to disclose their payments to doctors directly to the government, which will in turn put them on a public website that makes the data "downloadable and easily aggregated" -- that is, as structured data. (Speaking of disclosures, here’s one from us: ProPublica has been consulting with Deloitte, who is bidding on a contract to build that website.)

Drug companies and manufacturers will also be required to disclose, on an established timeline with specific definitions, precise payment amounts and to disclose a doctor's middle initial, if they know it. Companies' disclosures will also have to include each doctor's National Provider Identifier, which is a unique numeric identifier for medical professionals, though that number won’t be made public. So when PPSA is fully implemented in mid-2014, the payments will be comparable between companies and across time periods.

Until then, we’ll still be here doing things the hard way. Despite all of the problems, we believe strongly that Dollars for Docs is an important tool to ensure that Americans can have full and frank discussions with their doctors about their financial relationships.

One More Thing

Manuel Aristarán, a Knight-Mozilla OpenNews Fellow at La Nación, a newspaper in Buenos Aires, Argentina, has been developing a tool called Tabula for extracting tabular data from PDFs. Tabula's interface and algorithm for assembling characters into lines of text are much more developed than Farrago’s, so we’re helping fold Farrago into Tabula. Look for an open-source release of Tabula very soon.

Update: Tabula has been released.

No Windows. One Exit. Free Drinks: Casino-Driven Design for Crowdsourcing

2013-03-20T12:02:02-05:00

During the 2012 election, we created Free the Files, an interactive news application based on crowdsourced data, built in real time by thousands of volunteers. It was a collaborative effort to track TV ad spending by campaigns, super PACs and so-called “dark money” nonprofit groups in the country’s top swing markets.

Measured by participation rate, Free the Files was an astonishing success. More than 1,000 contributors submitted over 94,000 transcriptions to help turn messy invoices from local TV networks into clean data. One volunteer transcribed over 28,000 filings. Each transcription was “verified” after two or more users agreed on all of its data points. There are currently around 17,000 verified filings, and people are still working.

Much of this success came from the efforts of our expert engagement team, who motivated and interacted with our volunteers every day, and who hatched a clever campaign of social media, contests and promises of free T-shirts. Our users were, of course, also motivated by their own sense of civic responsibility, wanting to help build the first free database of political TV ad spending.

But design played a significant role as well. We kept our eye on optimizing each page for participation. We called the design we devised for participation-oriented areas of the site “Casino-Driven Design.” A variant of Behavior Design, Casino-Driven Design cuts away all distraction and drives the user’s attention toward staying focused on a single task.

No Windows. One Exit. Free Drinks.

Why do we call it casino-driven design? Casinos are notorious for adopting an interior design that keeps people gambling. There are no windows and no clocks so it’s easy to lose track of how long one has been gambling.

Casino-driven design creates an optimal atmosphere for task completion by actively discouraging cross-site exploration and page exits. And, like in a real casino, we keep the small rewards flowing, such as seeing your name on “freed file” pages and on a leaderboard.

There are no site-wide template elements on casino-driven pages — no section links, ads or even a link to the homepage. Casino-driven pages start out as a blank white page. We add only the elements necessary for users to understand where they are and the nature of the next task. The only way off the page is to complete the task or abandon it.

But there’s more to casino-driven design than just a clean page.

Glass Doors

There are two important goals in getting people to complete crowdsourcing tasks: Getting them to start volunteering, and getting them to keep going.

We tried out a bunch of strategies for getting people into the app for the first time, but our favorite was the “glass door”: We show a tantalizing taste of the activities that will become available after a user takes the first step — signing up:

Behind the “frosted glass” is a view of what you’ll be able to do once you’ve logged in. It’s hard to get to that screen and not want to see what’s behind it.

We also helped our readers find an activity using yellow and blue wayfinding at the top of each page:

Yellow status bars across all of our apps mean “this stuff is about you.” In Free the Files, we used it to give people quick links to logging in, an at-a-glance look at how many files they’ve freed, and what’s coming up. Light blue boxes and bars across our apps mean “here’s the most important thing you can do” (we call this shade of blue “do-something blue”). When we launched an election-day contest to transcribe all the files in Las Vegas, we put a blue bar on top of the pages to let them know there was an important activity to take part in. By the end of the single-day Las Vegas challenge, we had transcribed every file in the market.

Low Friction

Once inside, we keep the click targets really big and bright, and the number of actions as low as we can get away with. We spent a long time editing down the number of data elements we asked our readers to help us transcribe. In the end we pared down to four elements — and after launch we found ways to cut that down even lower.

Once the user has transcribed them, he or she can mash the big green or red button. Quite satisfying!

To keep people around longer, we engineered the page never to fully reload. There was never a reason for people to leave the page — we provided pop-up instructions and autocomplete boxes for values we thought users may type in. For example, we preloaded the “Who bought it” autocomplete field with committee names from both the FEC’s database and the FCC’s file naming scheme so users could start typing and choose one rather than try to decode the often confusing forms.

Finally, we borrowed a trick from the gaming world by adding a leaderboard so that users could (justifiably) brag about how much they were contributing. They became super-competitive about their spot on the board. During the election-day challenge, we showed users their single-day count right in the yellow bar on each page to help them keep track of their score.

Users were also able to brag in social media about freeing files — in a popup, without leaving the casino.

Casino-driven design was resoundingly successful in our election-related apps. We later used it to help drive participation in our Message Machine project. We’re excited to keep evolving it when we build crowdsourcing apps.

Looking Ahead

Casino-driven Design is all about reducing friction to participation. We showed Free the Files to some MIT computer-science profs and they gave us some interesting ideas we’re eager to try in future casinos:

One of the things that made transcribing the ad contracts difficult was that there were a plethora of proprietary page layouts that each station or network of stations used. This meant that users had to hunt around for the same data points in different places on different filings. It’s possible to cluster similar-looking documents using a technology called computer vision and then only show one kind of page to each user. This ought to speed their work up.

We also plan to experiment with presenting users with a single task — say, transcribing just the date on a single form design — and repeating the task over and over again. This may let the unconscious mind take over and speed task completion enormously. Different users would see different elements to transcribe, so we’d still end up with the same data, just split up more atomically.

Beyond that, we could also ask users simply to draw boxes around the spot where each data point can be found in these various formats and then write software to look in those places and use OCR to transcribe what it finds. OCR isn’t a good fit for analyzing entire documents like in Free the Files (especially scanned or faxed pages), but if we have human help to guide the OCR on these boxes, we may eventually be able to grab data out of forms with only human verification.

ProPublica Honored with Best Map, Two Medals at Malofiej 21

2013-03-18T11:03:05-05:00

On Friday, the Spanish chapter of the Society of News Design (SNDE) announced the winners of the 21st Malofiej International Infographics Awards. The jury evaluated over 1,000 entries from 28 countries.

This year, ProPublica was honored with the Miguel Urabayen Award for best map in the online category for StateFace, an open-source font we created. It’s made up of U.S. state shapes and is meant to be used as a design element in interactive web apps and graphics. The judges were impressed by StateFace’s versatility and its availability as a public service. Besides our own work, StateFace has been used by NPR, the Guardian, the Huffington Post and many others, especially in graphics and applications about the 2012 election. ProPublica also received a silver medal in the Innovative Format category for StateFace, and a bronze medal in Online Features/World and Nation for Housing Segregation: The Great Migration and Beyond by Jeff Larson and Nikole Hannah-Jones.

The Malofiej awards have been called the “Pulitzers for Infographics.” We’re extremely pleased to have been honored with three awards in the 2013 competion. Congrats to all of the winners.

Read our Day 1 and Day 2 impressions of the Malofiej World Infographic Summit in Pamplona, Spain.

Between Human and Machine: Thoughts on Malofiej 21 Day 2

2013-03-18T10:32:54-05:00

This is the second of two posts about the 21st Malofiej World Infographic Summit. Read the first here.

In one of the final sessions of the 21st Malofiej World Infographic Summit on Friday in Pamplona, Spain, Nigel Holmes noted the common theme of “hands” in the week’s presentations.

“We are humans, we are not attached to machines,” Holmes said as he presented a tongue-in-cheek look at “non-informational art” — fine art made by graphic journalists and artists when they weren’t making infographics.

As the conference was coming to a close, Holmes’ drawing attention to hands struck me as particularly insightful. One of the main undercurrents during the week was the tension between illustrative graphics, often borne out of photographs and sketches, and data visualization, whose raw material is often the very machine-based spreadsheet.

The presentations on Day 2 almost seemed to alternate between these two styles of information graphics in turn. The day started off with Juan Colombato from La Voz del Interior in Córdoba, Argentina, who explained his methods of creating graphics, starting with sketching, photographs, tracing Google Earth, research and reporting with experts.

The themes in Colombato’s presentation were echoed in those from Amanda Hobbs (the former Art Research Editor at National Geographic magazine) and Jen Christiansen of Scientific American.

In contrast were the presentations of Jan Willem Tulp, and Frédérik Ruys, who spoke of the process behind creating data-based interactive and motion graphics.

What struck me about all the Day 2 speakers was the emphasis on the data collection and analysis process, something you don’t expect to hear much about at a conference dedicated to presentation. The Malofiej awards are judged on effectively conveying information graphically, not necessarily on how the information was gathered or analyzed. Nevertheless, process was a big part of the lectures, and it was fascinating to hear the different approaches taken by different publications. Both the illustration-graphics and data-visualization camps shared similar methods.

Hobbs couldn’t stress enough that the illustrations in National Geographic were not artistic renderings, but rather reported works based on extensive consultation with experts. Hundreds of emails and sketches (some on Post-Its) of proposed graphical bits were exchanged in pursuit of accuracy. Likewise, Colombato said that in order to cut through the jargon one usually hears from experts, he would often ask how they would draw something if they had to.

Ruys, a journalist and designer focused on motion graphics, and Tulp, a freelance interactive developer and designer, talked about using data as a primary source, and the struggles to clean the data as well as to find stories in it. When describing one of his sources, a database of shipping routes, Ruys showed an example of dirty data in the “destination” column. Certain captains would leave destinations blank, and others would even try to be funny, entering values like “pirate bay,” or “wherever there is water.” He ended up throwing out that data point because it was too messy. When looking at the shipping data for an animated sequence in a documentary called “Netherlands from Above,” he also spoke of finding stories in spreadsheets. In the end, he decided to focus on the journey of one ship rather than showing the whole set.

Tulp analyzed Dutch elections data for a project called Close Votes because he was curious whether certain cities in the Netherlands voted alike. Through his analysis and resulting data visualization, he discovered that the Dutch “bible belt” all voted similarly, and even discovered what may be the most religious city in the Netherlands based on its extreme voting patterns. While he could have done more to highlight the interesting findings (the graphic was, as we say, all near, no far), it was a good example of data visualization as generative storytelling — graphics that actually generate stories themselves.

The tension between illustration graphics and data visualizations also dovetails with another axis at Malofiej: representative and figurative work versus abstract and geometric presentation of findings.

The best example of this split was a story Jen Christiansen told of a graphic she was developing on the expansion of the universe for Scientific American. Having been inspired by a 1998 minimalist scientific graphic by Bryan Christie, she set out to explain the phenomenon with a very elegant graphic of continually expanding circles (the observable universe) within a cone. After speaking with editors, she gradually came around to the idea of adding more representational detail, such as making the circles spherical and adding small illustrations of space phenomena. While the colors in the initial version were bright and primary, the final graphic took on the traditional look of space graphics — black background with spheres. After seeing the final product, I finally understood what the graphic was explaining. Christiansen called these trappings a “welcoming gesture,” and a “glass of wine before a challenging plate of tripe.” Easing readers into data through representation is something those on the pure data-visualization side can learn a lot from.

These axes were a useful lens through which to understand where the Malofiej contest judges gave awards. The New York Times — especially Archie Tse’s work, which won a Gold medal for Individual Portfolio — shows a mastery of both axes: illustration-based, representational explanations of how various Olympians win at their sports, alongside abstract (to the point of austere) shoreline-free flood maps for Hurricane Sandy coverage. His work is based on sketches and reporting, and it’s based on massive data sets. The Malofiej judges looked for mastery along both axes.

Correction, March 18, 2013: Jen Christiansen works for Scientific American, not American Scientist. The spheres in her graphic represent the expansion of the observable universe, not discrete planets.

Correction, March 19, 2013: Amanda Hobbs was formerly the Art Research Editor, not an Art Director, at National Geographic magazine. The role of Post-Its in her work has been clarified.

Outsider CAR: Quick Thoughts on Malofiej 21 Day 1

2013-03-14T20:28:01-05:00

This is the first of two posts about the 21st Malofiej World Infographic Summit. Read the second here.

Today was the first day of the Malofiej World Infographic Summit in Pamplona, Spain. It’s a very small (the whole attendee list fits on two double sided sheets of paper) two day one-track conference. The Malofiej awards are highly geared towards news graphics (mostly print, but more so on the web), so naturally I thought the conference talks would be, too. But there was a broad spectrum of speakers. The big guys were there (Wilson Andrews of the Washington Post and Graham Roberts of The New York Times), and art directors from magazines in Brazil and Russia, but also a few, for lack of a better term, data artists.

Two talks, in particular, stuck out to me as examples of what I can only describe as “outsider CAR.” Only after I tweeted this, did I realize the coinage.

But, here’s what I mean. Nicholas Felton — creator of the infamous Feltron Reports — discussed both his methodology and production strategies for the report in general, but specifically focussing on the 2012 report. What struck me about his presentation was that he was using the vocabulary of Computer Assisted Reporting, but was using data collection, analysis and presentation methods he had come up with himself. Obviously, the Feltron Report isn’t a work of journalism. But it’s close. It actually could be, if he took the data one step further and found and presented stories in it. But everything from “sampling” (filling out surveys in a custom iPhone app that pings him randomly) to the final product (often using stuff like multi-slice pie charts, force directed diagrams, and “ego-centric” cartography wherein maps remove all points of places he hadn’t been) are things that would never be allowed in a news environment. Felton said that 2012 was the first year he started “sketching in code” using Processing. In previous years, he would start out in Illustrator to create, say, a pie chart, and then circle back and fill in the data later. I’ll be curious to see whether next year, he veers more towards journalistic methods, say, bringing the data into R and looking for trends and correlations that could then be “reported” on. Or, maybe doing a third party edition to get a dispassionate report, rather than relying on “sampling” that is probably swayed by his own view of what will turn up in the product.

Another talk, by Stefanie Posavec, a freelance “data-related designer” (her term) focused on a bunch of work, most notably her gorgeous diffs of Darwin’s Origin of Species editions. Late in the talk, discussed a new project she’s working on for the V&A Museum, and in particular, a map she made of triangles between country capitals. She made the map by tracing Google Earth and bringing it into illustrator. While, obviously this also isn’t journalism, I think it’s another example of “outsider CAR.” In a news environment, she would be invariably steered towards GIS software, but instead she came to her own cartographic methodology. When an audience member asked why she didn’t learn to code, she joked about her own laziness but also said it’s kind of like the difference between a hand-knitted and machine-knitted sweater. As a purveyor of bespoke artisanal data, I object a bit to that, but I wonder what the piece would have looked like if she had, say generated the triangulated map in PostGIS. It would have been easier, but it would have probably come out a lot differently. People have quite different conceptions of laziness!

I’m interested to see what tomorrow’s talks bring. It’s an interesting conference, and I think those of us who are often directed towards the right way to do visualizations, cartography, data collection, analysis and cleaning in our newsrooms and at conferences like NICAR, can learn from outsiders who are hand crafting data according to their own rules.

Casey Thomas, P5 Resident

2013-03-14T17:47:00-05:00

The fourth P5 Resident started a project in the ProPublica offices today. He’s Casey Thomas from AxisPhilly.

AxisPhilly is a non-profit news startup in Philadelphia. Their mission is to “educate and engage citizens on topics of public interest while empowering them with tools to participate in developing and implementing change.” A big part of their mission is interactive news applications. In a newsroom of nine people, AxisPhilly has two news apps developers and a freelancer who works on data projects. Casey’s projects at AxisPhilly include a map of the effects of delinquent properties on the market value of nearby homes, as well as a map projecting the property tax changes at each address in the city.

Casey’s project at ProPublica is about the 23 schools slated to close in Philadelphia at the end of the school year. More details when it launches very soon!

Everything You’ve Ever Wanted to Know About Our News App Tech

2013-02-27T11:14:54-05:00

As newsrooms incorporate news application teams, one of the first questions they have to answer for themselves is what technologies to choose, and how to set up developer and web hosting environments that are sane and tuned to serve news apps. As part of her P5 Residency, developer Peggy Bustamante from Digital First Media's Project Thunderdome spent a few days mapping ProPublica's infrastructure. She's written a post that lays out some alternatives that came out of a discussion on the NICAR-L mailing list, and the answers that the Thunderdome team came up with.

What follows is ProPublica's advice on developer and server setups for news apps teams.

Step One: Pick a Server-Side Programming Language and Stick to It.

One of the early decisions you should make is which programming language and which development framework to use. Two prominent options are Ruby on Rails and Python/Django. There are other decent options like PHP/Laravel and even the Microsoft stack but if you want to share server-side code with other newsroom nerds (and you do) you're best off choosing between Ruby and Python.

The news apps world is split somewhere down the middle between Rails developers and Django developers, with the Rails developers seemingly concentrated on the island of Manhattan. Both frameworks really great, and they both inspire a lot of pride and trash talking among their adherents. Which one to pick is predominantly a personal preference. If you and your developers like coding in Ruby, use Rails. If they like coding in Python, use Django. Neither is an objectively correct choice. There are newsrooms who have built apps in .NET and in PHP, too. Within a certain bounds of sanity, my rule is that if you're going to be writing in a language all day every day for years, you might as well pick one you enjoy, and for which there's a decent chance you'll be able to hire experienced developers.

An important caveat: The one thing you shouldn't do is pick a language solely because of corporate standards or because "it's what the CMS is written in." News apps are a new kind of development for newsrooms, and they require a new, more nimble way of thinking. The benefits of corporate standardization are real, but they shouldn't dictate the tools developer/journalists use any more than business-side standards dictate how we report the news.

Once you've picked the language and framework, do your best to stick to them. Trying out new languages and approaches for each project increases your technical debt and complicates your life needlessly, and makes hiring more difficult.

Step Two: Assemble a Team

The news app cube at ProPublica is currently eight people, but don't think you can't get started until you've got a budget that big. We started out much smaller (at first just me, then three of us for a long while), and many small teams around our industry make incredible news apps.

I often hear when I talk to newsrooms trying to assemble teams that there is no good, affordable dev talent in their area. My response is always the same: They're looking in the wrong places. Here's why:

The talents needed to be a great news app developer form a three-legged stool. The first is the ability to do journalism -- The imperative to get the facts right, "editorial judgment," etc. The second leg is enough design acumen to make an interactive presentation that people understand and that tells a coherent "story." The third leg is the ability to write code quickly. I would argue that this third leg is the most teachable of the three. The fact is, the aforementioned development frameworks do a lot of the real heavy lifting, code-wise; the kinds of apps we create -- where the data is complex but the interactions quite simple -- tend not to need truly high-performance dynamic code, and server caches like Varnish make even slightly inefficient code fast enough to handle enormous traffic.

So while you may think that you need a developer with a CS degree, and you may think you're competing with Google for talent, in fact the first place you should be looking for talent is your own newsroom. Most newsrooms I know have graphics editors and designers who have been secretly writing website scrapers and using graphics frameworks like D3 for years. Graphics people have the journalism chops and the design acumen to make outstanding news apps and often just need to be given the space and time, and maybe a little training, to step into the role of news app developer. Then, as time goes on, you'll find that their success grows better than linearly as you add resources.

It's also very important that the people on your news app teams think of themselves as journalists. They should file FOIAs and make reporting calls. If they're working with a traditional reporter they should read drafts and see outlines and story memos right from the beginning. Most importantly they should report to an editor and not the IT department. IT plays an important role in your newsroom but news application developers are much more like reporters than they are like corporate devs. They should be managed the right way.

Step Three: Outfit Your Team

At ProPublica, we're all on Macs, because the Mac OS's unix-like underpinnings make it a pretty natural environment to develop code for deployment on Linux servers, and its popularity makes it easy to integrate it into corporate email systems, to buy Photoshop for it, etc. We know other developers who happily use Linux on their dev workstations. It wouldn't surprise me to hear devs out there using Windows, though I'll leave as an exercise for the reader how to up Windows for web development.

We each have two monitors: One is a nice bright, sharp iMac screen which is useful when you're staring at a screen for 8-10 hours a day. The second is one we call a CDM, a low-end "cheap Dell monitor" which is useful to look at pages as we're building them. It's a worst-case scenario end-user setup. You'll find that some designs that look great on a beautiful Apple screen will seem washed out and almost-invisible on the CDM.

We do browser testing using several virtual machines, one for each supported Internet Explorer. We support IE8 and up, though you should decide what browsers to support for yourselves after taking a good look at your analytics. All the virtual machines run on a server in our server closet. Until recently we used the free version of VMWare Server and existing Windows licenses, so the setup costs were pretty minimal. We connect to the VMs Microsoft's Remote Desktop Connector rather than running virtualization software on our computers. Another option is to use the Microsoft-provided virtual machines, which can be installed free of charge with ievms.

Step Four: Organize Your Team

While we frequently collaborate on news apps, we don't take the approach that the entire team works on one project at a time. One of us takes the lead on each project (you can tell who was the lead on a given project by looking at the bylines on the app: the lead developer is typically the first). We find that having one person who can take the time to read drafts, collaborate with a reporter, do any necessary research and reporting, and in general have the responsibility and vision for an app is really useful. Big scrums, while often very useful, can end up breaking up projects into such tiny chunks that journalistic opportunities are lost. Somebody doing data gathering and analysis should be thinking of story possibilities, and the designer should be very well steeped in the vagaries of the data, and the backend and front-end development work often flow together so much that a single developer should do them both anyway.

Naturally, not everybody is a true polymath and can do design, back-end code and journalism equally well. But each of us has strengths and we help each other. The stronger designers help the stronger coders, who help the stronger reporters.

ProPublica puts all of its code into source code management, which is an absolute requirement for development teams (and even solo developers). We use git. There are others, but the competition is essentially over and git won.

Using git enables us to work together on projects without clobbering each other’s work, to keep track of our changes and to easily roll back to previous versions when we need to.

While we're developing apps, we run them locally using Rails' built-in server. That lets us test and experiment with things outside production so there’s no risk of bugs or nonpublic material appearing on the Internet. We pass URLs to each other and to the wider newsroom using special local DNS records, so sending a test URL to a colleague is as easy as sending any other kind of URL, though of course they can only connect to our local apps while inside our offices.

Rails apps are pretty self-contained so this setup is pretty easy to manage. Some other teams use virtual machines to ensure a closer match between developer workstation and production server, but our system has suited us fine so far.

In our news applications, the data is often as finely honed as an artisanal cheese, so we tend to check even our data sets into version control, so we can version them and to make deploys easier.

When a ProPublica news app is ready to deploy, we use Capistrano to send our apps up to a production server. Capistrano is a ruby-based system (the Pythonic equivalent is called Fabric). It lets you specify exactly how your app should be deployed -- where on the server your apps are located, any web server config changes that need to be made to accommodate the code, and any special commands that need to be executed at any point in the deployment -- like cache invalidation, database re-seeding, restarting the app, etc.

Our deployment recipe is pretty specific to us, but Capistrano makes it easy to automate pretty much everything about your deployment strategy. It even backs up previous versions of your apps so it's dead simple to roll back in case something goes wrong with a deploy. If you want to enable less-technical people to execute some aspects of your deployment system you can explore Webistrano, which is sort of a web-enabled version of Capistrano.

Step Five: Set Up Your Server Environment

One of the most important choices you'll make is your server environment. I'm going to talk about Linux and as you're still reading I'm going to assume you've already decided on using Linux.

There are a bunch of options for web hosting but I'll talk about two: Real-server hosting and the cloud.

With real servers, you may have the advantage of existing IT infrastructure. You may already have a server room and tons of bandwidth, and adding a few web servers would only be an incremental cost. But adding servers is a slow enough process that scaling out to meet temporary demand (say, adding ten servers on election day that you don't need anymore the day after) can attenuate the benefits of self-hosting. Also, in my experience IT departments want a say over what gets installed on servers they're expected to maintain, and depending on your IT department that might introduce cross-departmental management problems.

Cloud hosting gives you incredible flexibility when you need on-demand server instances, quick scale-outs, rapid OS migrations, etc. They're not as fast as real servers, and the costs can scale quickly relative to real servers. A low-end cloud server can sometimes cost only a few dollars or less a day (you pay by the hour), but a cloud servers can get expensive on the high end. Cloud is best when flexibility is more important than raw performance. In terms of security, cloud and real servers are a tie.

ProPublica’s production server environment uses Amazon Web Services. We use a fairly plain vanilla three-tier architecture with a cache server in front of a few application servers in front of database servers. Some of our databases are hosted using Amazon's face-meltingly cool Relational Database Service. RDS runs just like a MySQL for us. It can also be a drop-in replacement for Oracle and SQL Server, though we don't need those. RDS does not support PostgreSQL, so for apps that require it we run it on a real EC2 instance.

For cache we run Varnish, which is a very high performing write-through cache. Varnish makes it so that each unique request only hits our backend servers once. It's so insanely fast that we once had an app get 2 million page views in a few hours and our CPU load average barely budged off of zero. I'm pretty sure Varnish contains alien technology.

Behind our Varnish box are several app servers that run the news apps projects. They are not directly accessible over the Internet.

Incidentally, ProPublica's systems started much more simply -- we started out with just two -- even from the beginning you'll want to keep your database on a separate server. For a long time we only had two Amazon servers, and we served a lot of traffic with just that. Our current setup can scale out incredibly easily by adding more app servers.

Alternatives

Peggy's post talks about alternatives to doing things this way, and there are lots. In addition to using real servers instead of the cloud, some teams "bake out" their work using dedicated computers and then upload the baked-out static files to a service like Amazon's S3. Static files can't crash and they can handle insane amounts of traffic. But there are big tradeoffs. If you need to store user input, or if you have lots and lots of possible application end points, or if you want to let users search for arbitrary terms, baking out might not be the best option.

There's lots more to it but this should get you started. If you have questions, the best place to ask them is the NICAR-L email list. News app developers tend to lurk on that list.

P5 Resident Researches Other Ways to Serve an App

2013-02-27T11:13:59-05:00

Peggy Bustamante is a news app developer with Digital First Media’s Data Team, and was the P5 Resident at ProPublica in January. She spent her time working with ProPublica News App Editor Scott Klein on mapping ProPublica's tech setup. Scott wrote a blog post about ProPublica's setup, and Peggy wrote this post, about alternative scenarios and DFM's own approach.

The first thing you want to do when you join a news apps team is build cool projects.

But if you are a new outfit, as we are at Digital First Media’s Thunderdome data team, there is one big step before you can get to that happy place of creativity and news: you have to set up an environment to build those cool projects.

My visit to ProPublica as a P5 fellow last month coincided with the crucial moment of our teams’ deciding how that development environment would be configured. On my first day, Scott Klein and I discussed ProPublica’s set up and what some other news orgs had chosen. It spawned a rather hearty discussion on NICAR-L. When the dust settled, we found some commonality among news devs and a variety of viable options.

Almost without exception, news apps teams are on cloud servers, not confined to building projects within the constraints of their news organization’s CMS. All agree, their work could not be done if they didn’t have a separate development environment.

While news app teams use a wide variety of languages, most use either Python/Django or Ruby on Rails. ProPublica and the New York Times use Rails. NPR and L.A. Times use Python/Django. DFM’s Data Team is starting with PHP because our developers have extensive experience building projects with it. In the coming months, we will be transitioning into Python/Django. We also use JavaScript and JSON feeds to build a variety of data-driven projects, such as our NFL playoff predictor.

ProPublica’s server setup, where the discussion began, is what I imagine is fairly standard, one that we as a team at DFM had been considering. It consists of a Varnish server in front of two development servers that are connected to database servers. ProPublica uses Amazon Web Services and Amazon’s Relational Database Service. They also have a second server for PostgreSQL because RDS does not support PostgreSQL. Lots more details on ProPublica's setup is over at Scott's post.

The Heroku Option

In the NICAR-L discussion, some recommended Heroku, especially Chase Davis, formerly of the Center for Investigative Reporting, and Ryan McNeil at Thomson Reuters. Chase Davis' blog post on Heroku is quite useful. Although originally only for Ruby on Rails, Heroku added Python support in late 2011.

One big advantage of Heroku is that there is no server configuration, as there is with AWS, and it’s easy to increase and decrease server capacity depending on a news app’s traffic.

On the downside, although deploying news apps is greatly simplified, Heroku can get expensive fairly quickly. It was generally agreed that Heroku is a great solution for smaller apps, but not for large-scale apps. A large part of the expense comes from using Heroku’s databases. CIR avoided that pitfall by using Heroku servers and putting databases on their Amazon EC2 box.

Heroku's costs can also be kept down by keeping the amount of data a news app uses to a minimum and putting in a little more effort with caching and configuration. That would allow an app with modest traffic to fit within the free account for a while.

But for news apps teams shooting for more complex apps with larger databases, another solution is...

Cooking Up Apps By “Baking”

The “baking” option, which is gaining ground, involves outputting or “baking” all the possible pages of a project to flat HTML files and then serving them from Amazon S3. The L.A. Times and NPR use this technique extensively. The advantages of this approach are that it is extremely inexpensive, and spikes in traffic aren’t a problem. And because there is no user input, security is much less of an issue.

Scott Klein, however, points out some of the disadvantages of “baking”:

Just to play devil's advocate, if your app doesn't have search or store user input, the whole thing will be in Varnish's cache (or your favorite framework's page cache) almost immediately anyway, so the resilience and performance benefits of baking out might be less than you think. And if you've got an app with lots of possible end points (say, millions of doctor payments or a national slippy map with 16 levels of zoom) the complexity tradeoff in baking out pages goes under water pretty fast. When you find an error in your map you don't want to wait three hours while your bake-out finishes to fix it.

The current rule seems to be whenever you can, bake it flat. Up to 80 percent of the projects at NPR are served this way.

What DFM’s Data Team Ended Up With

After much discussion and consideration, the DFM Data Team settled on our own version of all of the above.

As with other news apps teams, our development environment is separate from the CMS. We are also in “The Cloud,” albeit an internal cloud here at Digital First Media, which makes sense for us because the company has dozens of news entities that we serve.

Our configuration has load balancers in front of the database and production servers, which are fed by an identical staging server for testing and sharing works in progress, and a separate development server.

We have also decided to “bake” projects whenever possible, which will be even easier as we move to Python, so we can explore Ben Welch’s delicious Django Bakery.

And now we can build cool projects.

ProPublica’s News Apps and Data Guides

2013-02-27T11:06:04-05:00

Today we're publishing a series of guides that we hope will be useful for news app teams everywhere: A News App style guide, a high-level design overview, a coding manifesto, our standard social tags and a data bulletproofing guide. They represent what we've learned and our best advice for designing consistent, social-optimized and impactful apps in a sane dev environment.

See all the guides here.

The guides were written by my colleagues Jeff Larson and Jennifer LaFleur and me.

We're not publishing the guides on this site. They'll live on GitHub, a site that hosts open source code repositories. On GitHub, other newsrooms are welcome to fork our guides for their own use and to recommend changes to us, either using GitHub's "issues" feature or by forking, making a change in their own form, and submitting the changes back to us as a "pull request."

We mean the guides to be living documents. We'll be adding new things as we learn them, and we encourage everybody to submit what they've learned as well. Our opinions are strong but weakly held, and when we change our mind, you'll see changes in the documents.

Please let us know if you find the guides useful, and make sure to submit ideas via GitHub!

RIP EveryBlock

2013-02-08T15:47:27-05:00

With the closing of EveryBlock yesterday, both of the main reasons I started working in News Applications have disappeared from the internet. The other reason was a New York Times app called Represent, which allowed you to keep an eye on your elected representatives. The world is worse off with these pioneering news apps gone, but retiring old apps is something that our industry has to come to terms with. At ProPublica, we’ve started to retire our old apps, by removing search boxes and dynamic calls to the server. We’re making every effort to make them available on the internet forever. The process is a simple one:

wget -mkr -nH --no-parent -p --convert-links --content-disposition --adjust-extension http://url_to_app/

EveryBlock and Represent opened up doors for me: Represent got me started with GIS. The blog post about Represent’s stack is where I learned about GeoDjango and ogr2ogr, a tool I use every day. I even made my own geocoder following their lead.

Everyblock inspired me to write SimpleTiles after reading this blog post a million times, and I still use the bar charts outlined in this article for almost everything.

It is a shame that they are gone from the internet, and I hope we can replace them post haste.

A New Way to ‘Check In’ on Education Inequality

2013-01-24T15:19:54-05:00

Starting today if you connect your Foursquare account to "The Opportunity Gap," we'll send you stats about schools whenever you check into one. If you've checked into a school we've associated with a Foursquare "venue," we'll show you some details and give you a link to that school's profile.

You can also tap the ProPublica section of your checkin (see screenshot to right) to bring up that school's profile and compare it to nearby schools right from your smartphone.

A year ago when we launched the first version of our "Opportunity Gap" news application, we tightly integrated Facebook in order to make it easy for readers to compare schools and share their school comparisons. Today's relaunch adds Foursquare, along with adding a slew of new data to the app as well as algorithmically generate narratives by Narrative Science.

To accomplish the Foursquare integration, we're taking advantage of their new Real-Time API which lets us send push notifications in response to checkins. In order to associate schools with Foursquare venues, we used Foursquare's search API with its "match" intent -- a specially-designed endpoint for "venue harmonization" between apps. We ran our database of over 50,000 schools through Foursquare's API to store the venue IDs. If you check into a school that we haven't matched to a venue in our database, we'll use your location and the school name to show you a number of guesses as to what school you're at (we don't store your checkins or location data in our database at all). Once you pick one, we'll use your checkin data to link the school venue, so the next user that checks into that school will immediately see school stats.

Many news applications are location-based, and we're excited to start experimenting with bringing our apps to users where they are.

How To Edit 52,000 Stories at Once

2013-01-24T15:19:01-05:00

Today we're launching an update to our Opportunity Gap news application, with two new data points, better design on smartphones and integration with the Foursquare social network.

Also launching with this new release are short narrative descriptions of almost all of the more than 52,000 schools in our database, generated algorithmically by Narrative Science, a startup based in Chicago. Narrative Science launched in 2010 out of a research project at Northwestern University. Their platform uses artificial intelligence to turn structured data into human-readable narratives. For example, Forbes uses their technology to turn quarterly corporate earnings reports into short narratives.

We were introduced to Narrative Science by the MacArthur Foundation, which has supported ProPublica since our inception in 2007. In our initial conversations with the team at Narrative Science, we realized that ProPublica's data team and Narrative Science share a common goal: To make data tell stories. We also had a hypothesis, which is that adding narratives to each school page would provide an easier way for people who learn verbally rather than visually to understand the data.

Here’s how it worked, and what you might expect if you’re a news organization looking to experiment with algorithmically generated stories.

The project started with a call with Narrative Science to talk about what was important in the data. We told them that the “nut” of our news application was that all too often, school districts and states don't distribute educational opportunities to rich and poor kids equally.

We sent Narrative Science’s engineers a raw data set and walked them through the field layout and data gotchas.

Within a few weeks we started trading drafts – about a half dozen at a time -- in the form of sample narratives. We homed in until the narratives clearly expressed what the often-complex data meant. In addition to making sure our data was right and how we described it was consistent, we wanted the copy to tell the same story as our interactive pages did – a story not just about individual schools but how each school related to other schools in the state with different poverty levels. In that way, we worked with the engineers at Narrative Science much as an editor and reporter do.

We also had a few style issues, nothing that wouldn’t be familiar to editors and reporters everywhere. Smaller edits ranged from applying AP style to pruning redundant clauses.

One thing we learned was that mentioning a data point in a narrative made it seem much more important that simply including it on the page of an interactive database, so we spent time picking the right variables to “promote” to the narrative.

On to some of the more technical stuff: In addition to the data that’s visible on each page in the interactive database, our data includes some numbers used behind the scenes, such as calculated fields and an explicit pairwise relationship between similar schools. We found that it was easier to send those data points to Narrative Science rather than have them redo the often-complex calculations. Small differences between the narrative description and the graphic were cropping up, and that was the easiest way to avoid them.

Unlike a normal editing process, we weren't just working with a single story but tens of thousands of them. It isn’t practical to read 52,000 narratives. Also, Narrative Science’s systems are more complex than simple boilerplate with interpolated variables. Editing one narrative does not mean you’ve edited them all. In addition to recasting whole paragraphs, their systems generate a variety of sentences to express the same kind of data, so that reading the narratives for several schools would seem more natural and not automated. So edits that made sense in one case ended up not working in other cases, and sentences that seemed correct given one set of circumstances seemed wrong in others – often subtly.

We started getting larger samples of the generated narratives and we pulled random samples to spot check – just as we do with any data project – looking for problems with agreement between the narrative and the graphic, and for any confusing wording generated by the algorithm. Of course, if you see any weirdness – let us know.

Ultimately, Narrative Science delivered more than 52,000 narratives to us as json files over FTP, which we imported into our news application.

Whatever the future of the news industry holds, it seems clear that it will involve trying lots of things. And from our vantage point, algorithmic story generation seems like an intriguing solution to the problem of scaling narrative journalism in the era of big data. It’s helped us tune our data journalism for a new audience, and helped create stories at scale that would have been unthinkable otherwise.

New Year’s Resolution: Learn to Code

2012-12-19T14:51:14-05:00

A year ago I didn't know how to code. I had a journalism degree and had made some graphics, but I would have been hard-pressed to explain the difference between Ruby and JavaScript, and I was pretty happy when I got the YouTube video to embed correctly. I considered myself pretty technical but generally avoided the command line.

When I did start learning, I was amazed by how much was out there: introductory videos, explanatory blog posts, tips and tricks and step-by-step guides. If you're a journalist who wants to make a news app or a student interested in learning to code, you have plenty of paths to choose from.

To get coding quickly, nothing beats having a project – and a deadline. For some of my early projects (with the help of a few developers willing to answer my questions), I learned just enough to make it work: how to show/hide a div or generate a bar chart from a CSV file. Repeating those same steps for later projects helped me actually understand what I was doing (and how the code worked).

As the year went on I came across a number of resources that were also very helpful in digging deeper: tutorial videos that finally cleared up some long-standing befuddlement, random snippets of code that made the damn thing work after three hours of head-banging, some really good explanations I wish I had read weeks earlier, etc. I hope these links will be helpful to you as you get started coding. Most are Ruby and Rails related -- Ruby is a programming language and Rails is a "framework" that makes coding quicker and easier – with some JavaScript, Git, and other stuff that came in handy over the past year.

Ruby and Rails

Peepcode: Meet Rails 3 Part 1 and Part 2
Great introductory videos that are also very convenient – you can download them right onto your phone and watch them on the subway (if that’s your thing). Super comprehensive, might wanna watch them a couple times to get everything.

Rails Course from the University of Texas
A complete set of videos from a Rails online class from a professor at the University of Texas, a bit slower-paced and more detailed than the Peepcode lectures.

Rails Guides
This is one of the best resources for both getting an introduction to different components (like routes or active record, etc) and also something to look back on every time you forget the correct syntax.

Bastards Book of Ruby
For a detailed introduction to how Ruby works, from erstwhile ProPubNerd Dan Nguyen.

Ruby on Rails Tutorial
Super detailed, super comprehensive online book that walks you through the entire process of building an app from install to deploy.

RailsCasts
Screencasts for learning Rails. Tips and tricks, how-to’s for solving specific problems, and introductory “tours” of Rails techniques.

Rails for Zombies
A pretty entertaining way to get the basics of Rails down in a real “lesson” format (complete with quizzes and exercises and many zombie references).

More Ruby and Rails Resources

Ruby Programming Language official page
Screencasts from the Rails official page
Programming Ruby: The Pragmatic Programmer’s Guide
Learn Ruby with the Neo Ruby Koans
Learn to Program (ruby)
Rails API
Ruby Tutorial
Rake
Loops
Helpers
Link_to and css classes
Case Statements in Ruby
Strftime
Indexes in Rails

Javascript

Command Line

Meet the Command Line
Zed Shaw’s The Command Line Crash Course

Sinatra:

Rapid Prototyping with Sinatra

Git:

Get Started with Git
Pro Git Book

CSS

Colors

Regular Expressions:

Ruby regular expression editor
Regular Expressions
Learn Regex The Hard Way

Sublime Text

Sublime Text Tips and Tricks

Vim

Using the Vim Editor

Markdown

Markdown Basics