Joyent

The new library

Time was you did systems and networking work, you had The Library. Programming Perl , sendmail , DNS and Bind , TCP/IP Illustrated , UNIX System Administration Handbook . These books documented the core knowledge to operate effectively in our industry. You’d add others as your level of badassery increased, perhaps Mastering Regular Expressions , Internet Routing Architectures , or APUE , depending on your passions and your role.

Times have changed. You still need to have all the old knowledge, but you need quite a bit more to meet the bar in the modern, massive-scale, online services world. Thousands of servers, distributed storage, databases with billions of rows, real-time monitoring of it all. The books below should be in your library and considered core for you and anyone else in the field.

Scalable Internet Architectures by Theo Schlossnagle

Architecture is the most important thing in building and operating a scalable service, and this book is the best book around on the topic. Read it, internalize its message, build better systems. I’ve never met him, but all signs point to Theo being an awesome guy, to boot. His blog is here .

Solaris Performance and Tools by Richard McDougall, Jim Mauro, and Brendan Gregg

You’ve got your badass architecture laid out, code written, now what? You need visibility as far into your system as you can manage so you know the state of the world and where you can make the most impact in server and application performance. Enter DTrace. Yes, this is a book about Solaris, but DTrace is built into OS X and FreeBSD (and Apache, thanks, Theo!) now, as well. DTrace is a quiet revolution in system performance monitoring. Master it.

A couple of the authors blog often enough to matter: Richard McDougall’s is here and Brendan Gregg’s is here .

High Performance MySQL by Baron Schwartz, Peter Zaitsev, Vadim Tkachenko, Jeremy Zawodny, Arjen Lentz, and Derek Balling

Database tuning has always been a bit of voodoo to me, perhaps because most of my exposure to it has been ninjas like the DB architects at Amazon tuning Oracle to within an inch of its life. This book demystifies the space, and thank goodness, because you are going to need it to get the most out of your infrastructure. How can you go wrong with folks like Jeremy Zawodny? You can’t. Here’s his blog .

High Performance Web Sites by Steve Souders

You’ve mastered your architecture, instrumented your servers, and learned to tune the crap out of your databases. Now you can, and should, turn your attention to the performance experienced by your users. That was the whole point, right? Steve Souders created YSlow while Chief Performance Yahoo! at the purple giant, and is now doing similarly useful work at the GOOG. Follow him here .

The Art of Capacity Planning by John Allspaw

Let’s get this out of the way: I love this book! Allspaw knows capacity planning for online services because he lives and breathes it and his passion comes through clearly in his writing. Think “the cloud” world of dynamic, on-demand resources free you from having to do real capacity planning? Think again. Now you have to do it even faster and the flexibility of the new tools means you will discover and exploit all sorts of new capacity planning and management techniques. Get the latest from John here .

So, there you have it! Get to reading and then get to building.

Seven Lessons I've Learned About Startups from Drag Racing

I haven’t actually been drag racing myself. Rather, I’ve been a casual watcher of so-called “nitro-class” drag racing since it is one of the sports offered regularly on ESPNHD on many lazy Sunday afternoons. Watching nitro-class drag racing on television allows one to only gloss the depths of this sport. One must actually go to a race in person to understand the lessons nitro-class drag racing offers entrepreneurs in the software business.

1. A startup is like nothing else. Be prepared for the shock (and awe).

When I heard on one televised race broadcast that the NHRA (National Hot Rod Association) would be coming to Infineon Raceway, the race track in the county (Sonoma) just north of me, I bought a ticket. I drove up on a Sunday, eliminations day. The crowd was a NASCAR crowd and vast. I’d bought a seat that was 40-50 yards away from the starting line. The opening ceremonies included what we call in the software world “booth babes” and all the racers came to the stage to be interviewed. Then the stage sort of parted in two and two “top fuel” cars pushed forward and did a burn out. I’ve linked to representative video:

My head and chest cavity nearly exploded from the noise and vibrations. As I looked around I noticed nearly everyone had some sort of ear protection, the kind worn on the tarmac around commercial jets. The top fuel cars rolled back to the start line, went through a liturgy of staging, second staging, foot just off the accelerator, brake down, clutch about to be engaged, the racers’ eyes on a pole of light that finally goes green and: oh my God.

The best technology demo I’ve seen ever. These people had created something so powerful, so magical. My senses were excited and thrilled while at the same time being destroyed by the power of the cars themselves. This must be what Dante described when the purified souls ascend from the mountain of purgatory through a curtain of fire into paradise. And if you think I’m being over the top, I literally was covered in tiny black specks of burnt rubber that had been thrown into the air. I went through that fire.

The lesson? You’ll never do anything as direct and meaningful (professionally) as to experience and persevere in a startup. Your hands are on the controls. You get to create, shape, mold, direct. The people I’ve known to be unsuccessful in startups are those that wait for someone to tell them what to do. No, release the clutch, press down hard on the accelerator, explode off the line. When you’ve done the quarter mile you can huddle with the team to see what needs to be adjusted.

2. When you race, you move forward by winning. But you just have to win. You don’t have to kill.

I have only been a fan of the sport for less than a year, but it seems most races end with either one car exploding (blown cylinders, some catastrophic mechanical failure, lost of control and into the wall, etc.) or the race is down to the nose. We’ll deal with the first result in lesson three.

The down-to-the-wire result is what interests me. I think the same holds true for most startups. They either blow up (sometimes spectacularly) or they find themselves in long competitive slogs of back and forth down to the wire. The temptation in these kinds of races is to watch the competition, but that is a mistake. To many things are happening at a startup (or should be) for you to pay any attention to anything else but your result. If the machine is working (and it doesn’t always), the element that will win is your performance. But don’t push too hard or you might push your machine to the breaking point. This is the toughest skill to attain in a startup. I sometimes like to talk about it as “knowing the fire that’s going to kill you” versus “the fire that’s far away on the hill”.

3. Do everything you can to do one simple thing well.

There are two key components to top fuel drag racing. One is the performance of the driver, which is predicated on the other key component: the performance of the machine. In drag racing the one thing that the team does well, if at all, is to get the machine down the quarter mile track as quickly as possible without blowing up. When it works, it’s an amazing demonstration of technology. But it doesn’t always work. There are almost always some problems and the race quickly comes to its three second close. Things begin to fail. But that’s OK, you did one thing well and got the machine down the track.

How does a three second race have relevance for a startup which, haven’t we always heard, is not a spring but a marathon? Well, a startup is like a marathon in as much as you won’t get to an exit quickly. We’ll talk about this is lesson seven. But a startup is a series of sprints. Over and over and over. You have to be willing to recognize and respond to the many different opportunities that will come in the door. The is called execution. Do it quickly, increase your turnover.

4. It’s the hidden work that will make you successful. What’s going on in the pits?

It’s OK to experience explosions, too. I have noticed in top fuel drag racing even the winners sometimes limp to the finish line pieces of metal flying every which way from their machines. Now the hidden, but ultimately critical, phase of the competition begins. They have to push or tow that exploded machine off the track back to the pits where, according to the rules of the NHRA, they have 75 minutes to put it all back together for the next race. I don’t think I’ve ever got free from a 20 minute oil change place in under 75 minutes let along rebuilding the engine. This is something I clued into late in my visit to Infineon Raceway. About 55 minutes after the last race, the broken machines begin to roar, or more like throwing thunder bolts, off in the distance, off in the pits. It’s exciting: an explosive petrochemical call to the fans that they are going to see that machine fly down the track at over 300 miles per hour once again.

Startups have there pits and they are equally critical to a startup’s success. First, invest in customer support. The temptation is too skimp because it is not sexy (product development) or it is, on the other hand, painful and something to be avoided. I know I’ll get comments criticizing Joyent’s customer support. And that would be fair. We’ve had some spectacular explosions! But we’ve continued to invest in better and broader support. Customers will recognize this and repay you. Second, invest in metrics. Measure everything you can. This is how you know what to rebuild in the engine. Make measurement the operating mantra of the organization. Each person in your startup should have a small list of metrics for which she is responsible. And these metrics should roll up into one or two measurements the whole company can follow.

5. You don’t need VCs. You need sponsors. Sponsors are your customers/advertisers.

There’s been lots of conversation recently about whether to raise money from venture capital funds. It is important to make a distinction between VCs and investors. Venture capital is a type of model of investment you’ll rarely, if ever want to involve your business with. I think the drag racers have it right. For the most part they have sponsors, and when a racer wins they pimp there sponsors. That’s the ad-supported model. But is a customer supported model, not a VC Warbucks model.

What do you need VC money for? Equipment? Software licenses? Big salaries? You should be able to get a product done for sale within six months. So raise $100,000 to do that. And if the product doesn’t sell, well, that’s a catastrophic explosion. Try again.

Joyent operates many more than 2000 CPUs in our cloud and we’ve never raised VC. We scrapped. We used credit cards. Jason sweet-talked lenders. We got footholds and we exploited them. Yes, we have some initial investors, but they’re not VCs. We’ve talked to VCs. But it always turns out they want to push us this way or that. Besides, they don’t understand our passion: we build something called an Accelerator for paying customers, and we improve that when we experience explosions.

6. Join a team of racers. Partner with companies to expand your reach.

Before getting to the end, I want mention that most top fuel drag racers are part of bigger teams. This allows drivers to share resources, finds sponsors, cross-sponsor. For example, John Force Racing is a boot-strapped, very successful racing team.

For startups, the lesson is to partner. Not just with other startups, but also with bigger companies. This activity can be difficult, but must be the principle focus of a senior person at your startup. Fortunately, bigger partners are easier to get to than ever. They are watching blogs, conferences, and many other channels. Speak to them as you speak about your passion.

7. If you think drag racing is a lifestyle, you’re wrong. Startups are about a good exit (winning).

While I agree with David that startups should avoid VC, I don’t agree with the end-game of building a lifestyle business. Think like a drag racer. Win the race. Win the trophy. That’s an exit. Joyent has investors, and they want an exit. 37signals has investors. I’m sure Mr. Bezos wants an exit. Target your exit and achieve it.

The lifestyle? That’s being a drag racer. Hold the clutch just right, foot above the accelerator, breathe, watching the pole of lights, yellow, green. Go. At 300 miles per hour. Push the machine back to the pits to get ready to do it again. Next week onto another track, another startup. It’s got to be great.

Mi casa es su casa...most of the time. Joyeur commenting 'policy'.

Joyeur has a diverse audience with different sensibilities and opinions, and we tend to give folks a very long rope when leaving comments here on Joyeur.

Just please remember this is our home and we ask that you play nicely once you walk through the door. Good blog commenters add to the discussion and are known as knowledgeable, informative, friendly, and engaged. Bad blog commenters will be deleted, without notice, and possibly banned forever if you catch me in a foul mood. You can disagree with us, or any other commenters in this forum, but respect our space and keep your comments directed to the topic at hand.

To prevent your comment from being deleted:

• Please try to stay on topic as much as possible. We know sometimes you stray. That’s ok. Just try to keep it reeled in as much as possible.
• Relevant links in comments are okay, irrelevant links are not.
• Own your comment. We really don’t like ‘anonymous’ commenters and are more than likely to delete them. Be proud of what you have to say, and if you aren’t – then you should not be saying it. Linking back to your email address or website is even better as it shows me you are willing to stand behind your comment.
• No personal attacks, hate speech, excessive profanity, or other behavior that would be inappropriate in a civil conversation.
• No spam. We don’t like spam.

So don’t make us the bad guys. Just play nicely from the start and don’t track mud on the floor.

50 New Accelerators in One Go

I’m hanging out at the BuildingWebApps RailsQuickStart Seminar class in San Francisco.

Christopher Haupt just lead the class through the process of logging into their Joyent Accelerators for the first time.

This is the first time I have seen so many people log into new Accelerators simultaneously. Since we launched the Free Joyent Facebook Accelerators , there have been times when literally hundreds of new developers created new accounts in a day. But seeing it first hand is fun.

BTW, the next version of this class will be run on April 29-30. This class sold out, so make sure you register early for the next one.

Hello BingoDisk.

Just a quick note to confirm BingoDisk is online and good to go.

Dave will provide a summary post on what happened shortly.

The Systems team is focused on bringing Strongspace back online. Update coming.

Touch a Joyeur: Upcoming Events

CommunityNext (Sunnyvale, CA)
October 5-6, 2007
David Young will be on hand and leading a session on Friday (at 2pm) on ‘Building great businesses, not great datacenters’. It’s a barn burner folks – be there or be square. Register now.

Graphing Social Patterns (San Jose, CA)
October 7-9, 2007
Kristie Wells will be there on Sunday night and Monday, playing in the Open Faced event (Joyent is a sponsor) with David coming in on Tuesday afternoon to lead a panel on Hosting, Hustling, & Hog-tying Apps. Register now.

Boston Ruby Group (Cambridge, MA)
October 9, 2007
Luke Crawford will be giving a reprise of his packed RailsConf Europe presentation at the Boston Ruby Group meeting next Tuesday at the Harvard Law Library, Langdell South classroom. (Directions) The talk is entitled ‘Creating Hybrid Web and Desktop Applications with Rails and Slingshot’. Read more about it, and see you then!

OpenSolaris Summit (Santa Cruz, CA)
October 13-14, 2007
Catch Ben Rockwood and Jason Hoffman at the first ever Summit. It will be small (100 people). It will be informative. It will change your life (for the better). Register now as last time I checked, there were only 11 spaces left!

Web 2.0 Summit (San Francisco, CA)
October 17, 2007
Jason, Ben and Ian Murdock are joining forces to evangelize OpenSolaris and discuss ‘Why Operating Systems Matter’. The event is sold out, so if you have not purchased a ticket yet, we will try to podcast their session or will post their presentation online after the event.

Sun Startup Camp (New York, NY)
October 22-23, 2007
Sun puts on a great event for startups where you can learn from one another and get a chance to share your experiences while expanding your personal networks. It is free to attend, so if you are in the area – register now! David and Jason plan to be there and I expect will be leading a couple of sessions too.

Joyent New York Meetup (New York, NY)
October 23, 2007
David and Jason would like to educate you about the better bourbons in one of their favorite NYC bars. No tacos. But I would imagine a good time to be had anyway.

SNAP Summit (San Francisco, CA)
October 26, 2007
Mark Mayo will be on hand to lead a sessions around scaling Facebook applications where he will highlight the success of Are You Normal?. This Facebook application scaled to 100k users in 2 weeks using Joyent Accelerators.

Taco Tuesday (San Francisco, CA)
CANCELED
Sorry, Jason and Dave are traveling – Kristie is battling the flu bug. We need to cancel this gathering, but will be back on track on November 27th.

The benefits of IMAP and open: iPhone and Joyent Mail

Among the benefits of using open source (Joyent Mail) and open protocols (IMAP) is being able to use IMAP clients such as Outlook, Vista Mail, Thunderbird, Mail.app.

Why is IMAP great? Simply, it allows me to use email on different devices (a laptop, a desktop, a mobile device, a web client) while keeping all the devices in sync.

I’m very impressed by the IMAP client that ships with the Apple iPhone. Because it is a true IMAP client, unlike the behavior using a Blackberry, I’m able to read and respond to mail using the iPhone, file mail into folders, all the things that makes IMAP great. When I get back to using my laptop, IMAP ensures that everything is synched properly.

Now that Joyent Mail is open-sourced and free, we can work to make the workflow even better. For example, Joyent supports IMAP folders (some people think hierarchically), but also supports a rich tagging/keyword infrastructure (not yet supported in IMAP). I’d also like to see more work done to integrate IMAP with CalDAV (group calendaring, Joyent Calendar) with a view towards creating a true competitor to Exchange.

And the Joyent diet contest winner is...

...David Paul Young.

That boy lost 13% of his body weight, only narrowly beating out Pete who lost 12%.

It seems Jason, Shane and I never had a chance – though I would like to note, for the record, that I am down 6% which is not too shabby, especially as I have not seen the inside of a gym in almost two weeks.

We are also not exactly sure how Shane did since he didn’t show up for this morning’s weigh in so we went ahead and made the huge assumption he wasn’t even close, especially as he was just asking for ‘two more weeks’ a couple of days ago. :)

In total, the team lost a combined 77.5 pounds which is not a bad accomplishment. But then, considering we started off weighing a combined 1153.6 pounds, it seems maybe we could have done a little better with five weeks of healthier living.

So while it was hard to part with the $500 this morning, I must say it was money well spent as I got to sit back and watch Dave suck down cottage cheese and water in lieu of his preferred cheeseburgers and Manhattans. I am telling you. It was worth every single penny.

Round II anyone?

Speedo Tip Of The Week

Tip: If you want washboard abs and glutes you can bounce a quarter off of, you might consider working somewhere other than Joyent. :)

—————-

Of course, another route to bulging biceps and calves the size of baseballs is to simply put the Manhattan down and order a chef salad in lieu of the Mixed Grill which is what five of the Joyent staffers have decided to do, starting today.

Photo credit Michah

Dave, Jason, Kristie, Pete, and Shane have all kicked in cold hard cash ($500 each) for the bragging rights of who can lose the biggest percentage of weight in five weeks, with the winner taking the entire pot on June 28th.

And while it is no secret we have our fair share of ‘big boned’ people working at our little slice of heaven here, it was never more clear than at the weigh in yesterday. Once Shane took the bottled water and the roll of duct tape (cheater) out of his pants, we learned our combined weight totaled 1153.6 pounds. Wow. More than a 1/2 of a ton. Five (5) people. Not sure this is something to be proud of, but it surely is a conversation starter…

So, wish us luck – and please excuse us a bit if some of us get a little crabby over the next five weeks. Don’t take it personally, we are just carb deficient. And while some of the other (skinnier) staff are taking side bets to see who will wimp out first – I prefer to stay on the positive side of things. We need to do it. We have to do it. I mean, it would be nice if we could all get in an elevator together someday.

GoreRig(tm)

I got back from Railsconf yesterday and at some point picked up the latest Time magazine. It has Al Gore on the cover. There’s a picture of Mr. Gore working in his office. Did you notice the rig he’s working on? The GoreRig™. What struck me was the number of displays turned on. And the power they must consume. I like Mr. Gore. I was moved when I saw “An Inconvenient Truth”. I have biked more, taken the bus a few times, etc. because of the message of the film. But there’s something ironic about that picture.

Doing some back of the envelope calculations, I figure Mr. Gore is spending nearly $988 per year to power his three 30 inch HD Cinema displays, Power Mac, and the 32” inch (guess) LCD television (he’s not watching). I don’t have a television in my office. Just a 15 inch MacBook Pro, and a 20 inch Cinema Display. My rig costs $196 per year to power. Assuming each set up is on all the time, etc. etc. Like I said, back of the envelope. But that’s nearly an $800 per year difference. Seems significant. Especially if we all worked with the GoreRig™.

Here’s how I calculated:

I live in California where power is expensive. We pay roughly $0.15 per kW hour.

GoreRig™

1. Computer: Mac Pro 170W, ~5.9 hours (to consume a kW), $223 year. Source.
2. Displays (3): 30 inch HD Cinema Display 150W, ~6.6 hours, $598 year. Source.
3. TV: (guess) 32 inch Panasonic LCD 126W, ~7.9 hours, $167 year. Source.

Young Rig

1. Laptop: MacBook Pro 85W (when plugged in), ~11.8 hours, $111 year. Source.
2. Display: 20 inch Cinema Display 65W, ~15.4 hours, $85 year. Source.