When Lousy Code Strikes, Google Dispatches Its Elite ‘Gopher Team’
Brad Fitzpatrick was using the most routine bit of Google software you can imagine when the damn thing stalled. The download was 99 percent done and just stopped. Download stalls happen every day, but Fitzpatrick took it personally. He’s an engineer at Google, one of the most tech-savvy companies on the planet. This simply isn’t supposed to happen.
“It was embarrassing that Google couldn’t serve a 1k file,” Fitzpatrick said, telling the story last week at an open-source conference in Portland, Oregon. “So I bitched about it on our internal Google+ site.”
He was not alone. People had been complaining for years about the unreliability of the dl.google.com system, an invaluable bit of software that handles just about everything you might download from Google, from Chrome browser install files to the Android source code to tiny shared JavaScript libraries.
The problem was the software underpinning the file server system was more than five years old. It had simply languished.
“If code doesn’t receive constant love it turns to shit,” Fitzpatrick said.
The original C++ code wasn’t well documented, its automated tests weren’t up to snuff and no one really knew how it was supposed to work. People kept making incremental changes, resulting in a patchwork programmers refer to as “spaghetti code.” The unreliability was driving the server operations team crazy, but no one had time to rewrite something that technically worked. So Fitzpatrick volunteered to do it. That’s the kind of thing he does at Google, where he’s part of an team of about 25 engineers creating a custom programming language called Go.
“It took them a while to realize I wasn’t being flippant,” he said.
Fitzpatrick was only too happy to spend time re-writing something that technically worked — for another team, no less — because he needs guinea pigs. Fixing that bit of a code was a great opportunity to use Go and test it in a real-world system.
Go started as a Google-only project, but it’s been branching out. It was created by computer science veterans Robert Griesemer, Rob Pike and Ken Thompson. Thompson is the co-creator of the Unix operating system and the godfather of the C programming language. Pike worked with Thompson at Bell Labs. Griesemer is known or his work on Java compilers.
Go sought to combine the ability to handle a massive number of concurrent users — like the quirky telco language Erlang or the trendy server side JavaScript platform Node.js — with the speed of a language like C++. A distributed file server system that handles a massive number of simultaneous downloads from all over the world is exactly the sort of thing Go should excel at.
And it does, according to Fitzpatrick.
The new dl.google.com software contains fewer lines of code, uses less memory and, most importantly, is more reliable, he says. But he also learned a lot about the language along the way, and built a new open source caching library. He also discovered a few deficiencies in the Go language.
“I’m glad I was the first person to experience certain pain points,” he said in an interview after his talk at the O’Reilly Open Source Convention. The team was able to make many improvements to the way the language handles clustering and file transfers. The project was such a success that the Go team now regularly volunteers to help other teams with small projects. Even Griesemer pitches in. This “gopher team” has assisted the search team, the Chrome team and the customer survey team and others that Fitzpatrick isn’t allowed to talk about.
The gophers help out wherever they can, but they’re not advocating a complete takeover by the language. Some teams want to keep using the languages that they’ve been using. The teams are also sensitive to duplicate work.
“We try to use existing open source software whenever possible,” says Chris DiBona, Google’s open source manager. That means using the best software for the job, regardless of whether it’s written in Java, Python or something else entirely. That’s fine with Fitzpatrick.
“Our primary job with Go is make Google more efficient,” he says.
The language still isn’t widely used outside of Google, but it’s notched a few converts.
Former VMware CTO of cloud platforms Derek Collison believes the language is the future of the cloud. “The management layers and infrastructure layers of the newer technologies that provide this cloud delivery model?” he told us earlier this year. “Within two years, a majority will be written in Go.”
Couchbase, an open source database that powers applications for companies like Zynga and NTT DoCoMo, is written partially in Go. Its developers have also written parts of the system in Erlang, C and C++. But Go provides the best of both worlds, says Couchbase software architect J. Chris Anderson. “We’ve found that Go offers a concurrency model almost as powerful as Erlang, but with a developer friendly type system,” he says Anderson.
Collison’s prediction may be a leap, but his new cloud company Apcera is already using the language. And Go is, of course, making an impact at Google, where the dl.google.com system is a whole lot more reliable
