A commercial hosting company implements OpenBSD: AN INTERVIEW by Robert Bernstein PO Box 17312 Esmond, RI 02917 401-231-5502 poobah@ruptured-duck.com 2683 words For the past two years I have had an account on jtan.com, a web and Unix shell hosting venture owned and operated by Chris Nadovich since 1991. Chris is a practical guy. True to the engineering degrees (BS and MS in EE) he earned at RPI, Chris evaluates technologies on the basis of one criterion: does it solve a problem? So, when I got an email from him announcing that he was bringing online a new shell account host running OpenBSD 3.0, I sat up and took notice. I knew a couple of things about Chris, and about jtan. I knew he was scrupulous wherever security was concerned, so much so that when I recently took on an assignment to write a primer on host-based security, I asked Chris to review it. I also knew that jtan was hosting its shell accounts on Sun equipment, at least the command prompt in my shell showed: ~$ uname -a SunOS io 5.5.1 Generic_103640-24 sun4m sparc SUNW,SPARCstation-5 For these reasons, and more, it struck me that this move to OpenBSD was no casual event, and I wanted to know more. My first encounter with a *bsd system was with OpenBSD version 2.4; like many Linux users I suppose, when it came time to connect to a cable modem I chose a *bsd with Darren Reed's ip filter package rather than slog through the vagaries of Linux ip chains. Over time I have pretty much moved entirely to *bsd, and I'm typing this article on a _fairly_ up-to-date NetBSD (my kernel: NetBSD 1.5.3_ALPHA (BOB1) #0: Thu Oct 25 02:57:41 EDT 2001) while my current firewall machine is FreeBSD 4.4-Stable. (I refuse to choose sides in that debate!) I queried Chris about doing an interview for possible publication, and what followed from that request were some long thoughtful answers that provide a glimpse into the history of one corner of our neonatal Internet culture. Here are some excerpts from that exchange, starting with some personal background: RB. I see you have a Masters in EE from RPI. Were your EE interests then centered on computing per se, or on the DSP work that you still seem involved with, or on something else? CN. I was always interested in computers, back to about 1974 when I first learned to program -- in APL. But then I saw (and still do see) computers as a tool for solving problems rather than as an end. I don't tend to get into weighing the value of too many religious differences, methodologies, or styles. Just solve the problem. My undergraduate and graduate studies applied computers extensively, particularly in the fields of electronic and electromagnetic analysis and synthesis, as well as DSP, and "solving the problem" was always the bottom line. But I acknowledge that there are those that see computers as an end. Certainly when I graduated (1980) there then were many things called "computer companies" that saw computers as an end in themselves. These were businesses that actually made computers from scratch, and these computers were all different. It seems hard to believe now, but that was the way of things back then. On-campus interviews for EE grads were dominated by these companies, with names like DEC, Prime, Burroughs, Amdahl, Sperry, AT&T and IBM. In fact, in 1979 I went to a job interview for IBM in Boca Raton. They were developing a new "personal computer" that would be built with vendor parts and sold to commercial markets. They wanted me for a engineering position in the core technical group. I turned the job down, figuring that this "PC" thing was a toy and would never amount to anything. I don't regret that decision exactly, but I do see how ridiculous it may seem now. RB. How did you get started in the hosting business? CN. We started JTAN back in 1991. Eric Raymond was asking around in the Philly area to see if anybody had the resources to be a trunk site for Usenet. The company I was working for at the time had a pretty good net connection and he was wondering if they would share it with the community. They wouldn't. But the need was still there, so I decided to run a machine on my own to provide Netnews for people. Along with traditional UUCP newsfeeds, we also ran a BBS. An "advanced" feature of that BBS was a Unix shell account. People used the shell to read news and do email and later on to browse the web and chat on IRC. RB. Have you always used Sun equipment to host shell accounts? How did you first come to choose Sun? CN. Back in 1991 our first box was on Intel, believe it or not. Not SCO Unix though, we were running the newly amalgamated AT&T SVr4.0 Unix on an Intel 386/33. Youngsters might not realize that back then, a mere 10 years ago, Intel based Unix was very expensive, costing $2000 a seat. Unix had the reputation of being super expensive for everything, quite the opposite of today's perception. Both SCO and SVr4 were expensive, but SVr4 was the future and SCO was the past. The merging of BSD, SCO, and USL Unix variants in SVr4 was very controversial in the Unix community at the time, but in my opinion was long overdue. It gave us a widely supported and well documented target to work with. This was helped in no small measure by that wonderful book "Advanced Programming in the Unix Environment" by the late W. Richard Stevens. So we had a system. But once we got into providing netnews, as anybody who has done it will tell you, no amount of hardware is ever enough. Soon that 386/33 with it's "massive" 300 Meg disk was overloaded by Usenet and we had to expand. We still liked SVr4 and this is what led us to Sun. Sun had adopted SVr4 under the name "Solaris 2.0" and in the early 90's or thereabouts we noticed that low end Sun equipment running Solaris 2.4 was available at prices that rivaled equivalent Intel hardware. I say "equivalent" in terms of MIPS and Megs. There's nothing "equivalent" about Sun hardware when compared to typical commodity PC hardware. To say they're equivalent is like saying a Humvee is equivalent to an Escort just because they have the same top speed. Sun hardware is built quite a bit better, hardware wise, than a typical consumer PC. Given this fact, in the mid 90's low end Sun hardware became a better deal from our perspective than anything we could buy in the PC world, particularly if you wanted to run a mature SVr4 flavor Unix on reliable hardware. It seemed a no-brainer to expand by buying Sun equipment, like SS5's and Classics. Solaris 2.4 was almost identical to the SVr4.0 we had been running on Intel, so we ported over quite easily. Soon we were running several Sun boxes with Solaris 2.4. The 2.4 release of Solaris still had some "issues" which were mostly fixed in 2.5.1 around 1996. The 2.5.1 release of Solaris was very nice and we soon standardized on that. When 2.6 came out, and 2.7, and 2.8 we shrugged our shoulders. There didn't seem to be anything worthwhile in these releases for a public system that didn't have a frame buffer, not to mention that there were now some good Intel based alternatives. RB. As to the 'open source' unices out there, have you looked at the other bsd's (Free-, Net-), or at Linux? Did you install any of them? CN. We were aware of the BSD on Intel variants that were sprouting up in the second half of the 90's, particularly through our affiliate provider LineX Communications in California. They were very big on BSD, staying with the more BSD-like SunOS on their Sun boxes, running several NeXT machines, as well as some BSD on Intel boxes. Even though we were aware of BSD as a growing force in PC Unix, we were swept along with the Linux fad and we started to bring Linux boxes online without any serious consideration of BSD. That's not to say that we didn't consider BSD at all. It's just that, besides the small differences, that for some zealots I realize are all-important, there were no compelling differences for our particular applications that we could see at the time. It was the rise of evil in the networking world that opened our eyes to some "compelling differences" and eventually brought us to OpenBSD. RB. Did you consider other commercial unices? Perhaps the commercial bsd offering, BSD/OS? CN. After the mid 90's, in my humble opinion, there no longer seemed to be any point to any commercial OS for us. I know there are people out there selling commercial operating systems, and I hope all our competitors are buying them. I understand that some people may still pay $2K for a copy of SCO, Unixware, or even Windows. I hope what they are buying works for them. RB. What factors carried the most weight mitigating against continuing to use Sun? CN. As I mentioned, we began as a Usenet site, and we have always tried to keep up with that game. But Usenet demands more and more and more MIPS every year. In the last third of the 90's we needed to upgrade our news machine yet again, and this is what led us back to an Intel based OS. Thus, at some point we decided to build our latest and greatest news machine using Intel. For the OS we chose Linux -- more because it was known to work and we had it laying around than from any conscious analysis. Netnews endlessly demands hardware upgrades and while our ordinary shell/mail/dns services could run on the low-end Sun hardware, news could not, and when it comes to maximum MIPS for the buck, nothing beats commodity Intel PC hardware. The OS seemed irrelevant at the time, so long as nntpd would run on it. Now, of course, I would qualify that opinion. In any case, the equation had flipped around, and it seemed more economical for us to run on Intel than on Sun for high load service. Thus, we found ourselves running netnews and a few web-servers on Intel, but most of our other low-rate infrastructure on Sun. We would have stayed that way forever, maybe, but the world suddenly got evil and knocked us out of our rut. RB. How did you satisfy yourself that OpenBSD's claims as to security were _not_ hype, or PR? CN. Hype is something that flourishes better in the over-funded commercial world. I don't think a few prickly fish constitute hype. In any case, it was OpenSSH that led us to believe that OpenBSD was also useful from a security perspective. We had purchased a commercial license for F-Secure SSH to run on our shell machines, but we had endless problems getting it to work just right and we could never get support from F-Secure. I always hear commercial software salespeople going on and on about how commercial software comes with support, etc.. and that supposedly is a killer argument for any business to turn away from free software. What nonsense! Number one, a software customer wants the product to just work -- and there is no number two. I don't care if you have 1000 people waiting to help me at the other end of an 800 number, if the code is broken, I don't want it, free or otherwise. We quickly learned that the commercial SSH flavor was somewhat tainted with issues, whereas OpenSSH just worked. The recent exploits of the CRC32 attack fix illustrate this quite vividly. That exploit was known the better part of a year before some versions of SSH were fixed. A lot of people worked overtime to restore machines in the last couple months because of that bit of laziness. OpenSSH always seems to stay on top of these things, and in many cases hasn't been vulnerable to begin with. In any case, it was our positive experience with OpenSSH that led us to trust OpenBSD. RB. Any surprises during the beta test phase of your OpenBSD implementation? How long did it last? CN. I was pleasantly surprised with the /usr/ports tree and other upgrade capabilities. So far this has worked out quite well for us, simultaneously managing installed applications while giving us source-level control that is not available with RPM's. It also gives us "another fellow's opinion" about the security aspects of an app. If it doesn't exist in the ports tree, that serves as a warning flag that it might have issues. Authentication flexibility is another big plus that we found. Other than the usual "What is it this machine is supposed to do?" related head scratching, we pulled the machine together quickly. It installed, it detected the hardware, it worked. What else is there to say? We tested for about a month before putting any number of users on it. So far, user response has been 100% positive in the sense that not a single person has said anything to the effect "Linux/Solaris was better in such 'n such a way". Most of our users haven't even noticed the change. In our business, simply getting no complaints is equivalent to a rave review. RB. Ok, let me cut this litany short. The 'hot' questions here boil down to just two: Why not Linux? Why OpenBSD? Anything you could say on those scores would be appreciated. Yes, we are tip-toeing into potential religious war territory, but I intend to avoid submitting a partisan piece of propaganda... CN. I believe it was Christmas Eve 1996, or maybe '97, when I got a call that there was something wrong with the Linux news server. Upon investigation we found the news machine had been totally compromised. Most of the key system binaries had been replaced with trojanized versions and the machine was busily sniffing passwords off our wire, storing them in a file and posting them on IRC. We traced the break-in back to one of the many RPC related holes that existed back then in Linux. Well, all you have to do is spend one 24 hour session restoring from an attack on Christmas Eve to know that the network is no longer a safe and happy place. Shortly thereafter, we saw break-ins to our Solaris boxes. Since we were running eCommerce applications for ourselves and for our customers, none of this was amusing at all. We absolutely HAD to do something to try to fix this. Obviously we forced ourselves to get smart and get conservative in the services we ran on boxes, doing audits, setting up IDS, strengthening password management, etc.. But it seems to me that isn't really enough if the underlying OS itself has never been thoroughly reviewed. We DON'T have the resources to do an audit of Solaris, or Linux, or any other OS. We have to trust somebody to do that for us. And given that - OpenSSH had an excellent record at our site - OpenBSD had a plausible reputation for security - Nobody made a security argument to us to use any other OS - Linux, particularly RedHat, were targeted platforms for root-kits - Solaris 2.5.1 was getting out of date - commercial security vendors had uniformly disappointed us we saw OpenBSD as a possible way to give our customers more value, and to help us spend more time with our family during the Holidays. We know that no OS is bulletproof. Break-ins happen. But the 'home court advantage' of OpenBSD seems to be quite significant. Some of the things I say I know are contentious. I believe what I say is true, but I'm not an expert on BSD history or analysis. I'm just a humble businessman trying to make an honest living selling Internet services without getting pillaged by script kiddies. I want a platform that just works. RB. Thanks Chris; I think you've given our readers something to chew on!