This article put up with the permission of my friend Girish Venkatachalam. He has written quite a few articles including some for the Linux Journal. Below are some links to his articles: http://wiki.openbsd.org http://linuxjournal.com/9004 (Cover story of Aug 2006) http://linuxjournal.com/8289 (first international article) http://www.openbsd-wiki.org/index.php?title=Multiboot http://www.ciol.com/content/search/showarticle.asp?arid=23466&way=search http://www.mindtree.com/kc/view_art.php?wid=8 http://www.mindtree.com/kc/view_art.php?wid=9 http://mouthshut.com/user/girish1729/1.html Sandboxing and Virtualization demystified III Today we will talk about the various other sandboxing techniques and a little bit about emulation. Not so much about virtualization though you will often find these techniques used in the context of virtualization. We will specifically target the following techniques. a) chroot b) linux-vserver c) freebsd jails d) user mode linux e) Bochs f) wine g) Linux emulation under NetBSD,OpenBSD and FreeBSD h) kexec i) multiseat configuration There is enough on our plate now. Let us get to work. chroot(2) and chroot(8) have existed in UNIX since time immemorial. But I never so much as cared to figure out what it is about. Change the root? What is that? Why would one want to do that? But come virtualization and suddenly chroot is so popular. Actually it is a very high level of virtualization. It is the most popular and easiest sandboxing technique. It doesn't virtualize anything actually. But you build a bunch of tools around it, modify certain existing UNIX tools and lo, you have a sandbox that behaves more or less like a virtualized environment. Perhaps application level virtualization, but I will avoid polluting the term further. First what are the problems with chroot? Before that, what is it? And why is it so powerful? Let us start from a real world need. Easy to relate to. I have been debugging certain really hairy problems with FreeBSD ssh. I tested the client ssh and found nothing wrong. So next I wanted to test sshd. But sshd insists on having access to the key files, DH param files etc. and it expects it at standard locations. Though I could modify the source I could not get further as it is not at all prudent to diddle around with the source of security software. It was on a live production server. The best idea for me would have been to chroot to a temporary directory and populate the /etc and other directories relative to that artificial root directory and then run sshd on a different port. Why should you run sshd on a different port? Because chroot sandboxes only the file system, not network ports. Everything else needs to be done by hand. And it has the issue of being able to access open file file descriptors that are stored outside the chroot jail. As you can see there are plenty of problems with chroot. But it serves nicely for several real world needs. An attacker who breaks into a chroot cannot in general get into the rest of the system. And you can easily virtualize (in a sandboxed environment) the most common services like ssh,http(s),smtp etc. The fact that the kernel is same for all chroots is a matter of concern. And the rest of the resources needed by UNIX programs like uid, device files all these have to be taken care of. The main problem is in accessing shared libraries at right locations for dynamically loaded executables say bash shell. Without copying the libraries to the new root, it will not work. The best idea is to build a static bash. Assuming you have that, chroot is very simple. # chroot /tmp/chroot /bin/bash # mount -t proc proc /tmp/chroot/proc These two commands will give you a nice bare bones chroot environment. In fact this is the strategy to be employed for mounting linux root partitions on a running machine or running 32 bit executables on 64 bit PCs. And of course linux emulation on NetBSD,FreeBSD and OpenBSD. With all its defects,chroot forms the basis of sandboxing in the UNIX world. Now let us talk about the linux-vserver project. This is evolved from chroot's problems and it virtualizes a great deal of stuff. A lot of effort has been put into separating and isolating processes running inside different vserver instances. This is in fact a very usable and mature technique used especially in Debian based servers for server provisioning. Call it a poor man's virtualization strategy. But I mean nothing negative here. For most common uses linux- vserver suits just fine. Since Xen would be an overkill under many circumstances. Isn't variety the beauty of life? The main advantage of all these chroot techniques are amazing performance, scalability and ease of setup and maintenance. You even have different root users, different output for ps, scheduling done taking cognizance of this fact and various other tricks. FreeBSD jails are another effort build around the chroot technique. It is an advanced technique similar to linux-vserver. However I have experience with neither, so I refrain from commenting further. Welcome to the fascinating world of user mode linux affectionately called as UML. This is true virtualization/emulation unlike the above alternatives. However it incurs a huge overhead since it runs separate kernel instances. If not for that, would you call it virtualization? There are plenty of practical uses for such a thing and a lot of water has flown under the bridge since it was first released few years ago. The main line kernel has support for it and UML makes a great debugging and testing environment. Not only for kernels, but also device drivers and several other user land applications of course. Once you compile a user mode linux kernel using the appropriate make switches, you can easily start the brand new kernel just like starting a user level process. But bear in mind that user mode linux does not initialize hardware the way a stock linux kernel does while booting and hence simply because something works inside UML does not guarantee it will work directly. However most bugs can be tracked down and isolated using UML. It obviates the need for kgdb and a serial line and another computer to debug kernel code. Perhaps kernel programmers will know much more about UML than me. Bochs is the grand daddy of emulation as Qemu derives a good part of the code from Bochs. Its goals are to emulate the entire PC in software. You can think of it as a less feature rich Qemu. We must understand the fact that Bochs has been around for a while and that it has done some pioneering work in the field. Though Bochs is useful for running old DOS applications it has largely been superseded by other modern tools since the virtualization arena got really a great deal of attention lately. No article on emulation will be complete without chronicling the superb effort that has gone into the making of wine(No, not wine in Marcel Gagne's terms as I am a teetotaller). Wine Is Not an Emulator. It is an easy to use emulation technique that helps us run Windows applications from inside the cool comfort of Linux. However I cannot emphasize enough the fact that wine is not yet production ready. If you want to run a one off Windows application then wine is perfect. But for more serious use I suggest you use one of the other alternatives. One possibility if you have a LAN is to rdesktop to it or VNC to a Windows machine if not interested in getting into VMWare of Xen. But as far as my experience goes I have never missed any Windows application. My TV card was giving me trouble but today it works better than company supplied drivers! So I bid goodbye to Windows quite a long time ago. It will certainly interest you to know how Wine achieves its goal of emulating Win32 APIs on linux. Wine does something similar to Qemu for translating Win32 API calls of Windows executables to corresponding library/system calls in linux. However there the similarity ends as Qemu and Wine are in completely different problem spaces. Qemu emulates hardware and runs the Windows kernel(if you can call it that) and applications run on top of it and quite obviously you need a Windows license. But wine avoids all that. Considering this I think we have to certainly appreciate the efforts put in by Wine programmers as most of Win32 APIs are not even documented and Microsoft has this dirty habit of changing and introducing new APIs all the time. It is not always nice to play catch up game. Good. Now that puts things in perspective. Let us wind up with few more interesting things and then we will call it a day. I have explained the need for running different operating systems and emulation and virtualization. But I never suggested it as an alternative to porting software. This is where the excellent linux emulation support found in the BSD world comes in. It is often not very hard to port applications between POSIX compliant operating systems. But there is a case where this will fail. When does that happen? What about binary proprietary software? It is not fair to expect one to install an operating system just to run a piece of proprietary software. What to do? Run them under Qemu? Well, in case you have a fast enough machine, yes. But what about Skype eh? Would it work properly? I don't know. Anyway the nice folks(our friends) in the BSD camp have built an emulation layer on top of their kernels to translate linux system calls to run on their operating system. Now, this emulation layer is very very thin as it only emulates the ABI, the application binary interface which is a bunch of calling conventions; passing parameters and return values. Hmm, if it were so simple, I would not be talking about it. There is a lot of difference between the operating systems, even between OpenBSD and FreeBSD kernels. And there is plenty of difference between linux and all of the BSDs too. So things don't always work as expected but it is quite mature and usable in many cases and the performance impact is very minimal. So it is somewhat misleading to use the term emulation. As that term is mostly used to signify slowness. Most of the issues are related to the chroot technique like mounting the proc file system, creating device files and populating linux shared libraries. Of course the binary formats are different between operating systems and the loader figures out the magic value in the ELF header and translates/emulates transparently. It is really cool. When I was testing a NAT traversal program which was supposed to work on linux and FreeBSD, I just scped it to FreeBSD and ran. Really cool. And when my linux box got hosed due to hardware issue I used Skype under FreeBSD to call my friend in US. It performed very well considering the fact that my machine was a measly 233 Mhz box. Now coming to kexec. This is a very interesting concept. I am talking about it because it is somewhat magical like virtualization. The idea is reducing machine start up time or boot time. On several high end hardware machine boot time can be significant. And especially so in case of linux. BSDs are very fast when it comes to booting. Sorry I had to say it. kexec is a technique to simulate the BIOS in such a way that you avoid the need to reboot the machine with a power cycle to boot into a new kernel. This is a very interesting project and I am not sure how stable it is. What is multiseat computing? All our efforts so far has been to squeeze out the maximum possible from existing hardware by using software techniques. Isn't it logical then to speak of a way to share a single CPU between two or more terminals in a manner that is oblivious to the user that they all share a single CPU? Sounds like deja vu from mainframe world? Not exactly. By the way, xen is also an old IBM mainframe concept but carried out very differently. A mainframe terminal is just a terminal to access the mainframe. Whereas we are talking about full fledged computing resources at the user's disposal. Each user has an individual monitor connected through different VGA cards on the PCI bus and keyboards and mice daisy chained through the USB bus. This would make a lot of sense for supermarkets where individual computers are a waste and moreover it adds to electricity bills, noise etc... and needs extra space. However the problem with this is that VGA cables cannot be extended much. Now the technical scoop. X server has always supported multiple display sections and multiple input devices. So configuring X will do the trick. But the problem is in separating the keyboard and display inputs. Since currently if you connect multiple USB mice or one USB mouse with a PS/2 mouse, it ends up in a mux(multiplexer) and you can copy with one mouse and paste with another. We want to be able to separate them and assign to individual terminals. And of course this will not work for heavy duty computing. Theoretically it sounds rather difficult but practically it works well thanks to the efficiency of linux. Resources :- 1) http://www.ubuntu.com (Multiseat Ubuntu) 2) http://userful.com (Userful commercial multiseat) 3) http://www.linux-vserver.org (Linux vserver) 4) http://user-mode-linux.sourceforge.net (user mode linux) 5) http://www.ibm.com/developerworks/linux/library/l-kexec.html (kexec) 6) http://www/winehq.com (Wine) 7) http://bochs.sourceforge.net (Bochs emulator) 8) http://en.wikipedia.org/wiki/Chroot (wikipedia article on chroot)