Author Archives: ChipX86

About ChipX86

President/CEO of Beanbag, makers of Review Board, RBCommons, and Splat. Former old-timer at VMware responsible for Workstation, Player, Server, WSX.

WSX: Virtual Machines in Your Browser

Updated March 13, 2012 at 10:43PM: I have a list of limitations and known bugs toward the end.

Updated March 14, 2012 at 9:30PM: I’ll be on the VMware Community Roundtable podcast at 12:00PM PST today, where I’ll be talking about WSX and jumping into a little more detail.

Updated March 20, 2012 at 10:15PM: A lot of people are confused, so I want to point out that WSX is not AppBlast. That’s a separate project with separate goals. WSX is a remote console built using the same underlying technologies as Workstation and Player. It’s also not a front-end for View. I can’t speak for any of those projects’ goals and plans.

Virtual Machines have always been a great way to work with different operating systems, carry your desktop around with you, and manage lots of servers or configurations. In the past, you’d run the virtual machine on your computer and then use a product like VMware Workstation or Player to interact with them.

In Workstation 8, we introduced the ability to share VMs across a network with other copies of Workstation, and to use VMs running on ESXi/vSphere. You could use any computer in your network to reach any other VM and to manage your servers. I covered this in a previous post, and as I said there, we were very proud of this release.

Still, the world is evolving fast, and more people are moving to tablets and smart phones. It doesn’t mean the end of desktops or servers, but it does change how people are accessing their data and applications. And their VMs.

Get to it already

Alright.

I’ve spent the past few months on a prototype, one I’m proud to say we’re shipping as part of the Workstation Tech Preview. It’s currently called WSX (name may change in time), and it brings your VMs to your tablets, smart phones, and any PC or device with a modern browser.

WSX is installed as a mini web server in your network and serves up an interface for accessing your Workstation Shared VMs and your VMs on vSphere/ESXi 5. You can power your VMs on, off, suspend them, and interact with them. All from a web browser, and all without plugins, with nothing to install on the client end.

This means that you can walk into the Apple store, pick up an iPad right off the shelf, and in less than a minute, start using your desktop back home. (Of course, provided you’ve port forwarded your WSX server so it’s accessible outside your network.)

How does it work?

WSX makes use of some modern web technologies, such as HTML Canvas and Web Sockets, along with a small but powerful server to turn your browser into a full-on remote console. The WSX server talks to your Workstation, ESXi, and vSphere instances and relays the appropriate data up to the client running in the web browser. With that data, the client can stay updated with the latest changes to the VM and offer a full display of the console. We don’t use any plugins, meaning there’s nothing to install.

It’s known to work with the latest versions of Chrome, Firefox, and Safari. It also works with the Internet Explorer 10 preview (though I’m still working on some bugs there). And for tablet users, it works quite well with the iPad running iOS 5+. Android users running Ice Cream Sandwich may get some luck with Google Chrome for Android, but I’m still working on Android compatibility.

A lot of work has gone into making this pretty fast. If you stream a 720p YouTube video inside a VM and access it from Chrome or Firefox on a modern PC, you should see near-native quality and framerates. It’s not as fast streaming to an iPad just yet, but you’ll see some impressive changes there before long.

On some modern browsers, you can even make your desktop go full-screen, just like you can with Workstation.

Note: There’s a bug on the iPad today with the on-screen keyboard where capital letters and punctuation are a bit broken. I’ll fix it!

Tell me some use cases

Sure thing.

  • You can work on your documents from your iPad from anywhere, knowing your data is safe in your network and not on some desktop streaming service company’s VM somewhere in their datacenter, not worrying about what may happen to your data if their service is down.
  • If your server is acting up while you’re at the store, you can connect to your VM from your phone/tablet and deal with it instead of rushing home.
  • Play Windows Solitaire from your iPhone.
  • Confuse people by running a VM from within a browser from within a VM from within a browser from within a VM from within a browser from ……
  • Have your presentation or demo live within a VM so that when your laptop dies at just the wrong time, you can grab anyone else’s laptop or iPad and quickly resume where you left off.
  • Have your iPad set to sync music over the network with a copy of iTunes running in a VM, and then use the same iPad to interact with that same copy of iTunes without ever involving a PC or Mac.
  • Run Windows 8 in your VM full-screen on your iPad, so you can confuse everybody.

As WSX evolves, so too will the use cases. I’m personally very curious to see how people will be using it.

Installing WSX

Right now, WSX ships with the Workstation Tech Preview for Linux installer. You’ll be asked for a port (defaults to 8888), and then it’ll install once Workstation is installed. Make sure you have the python2.6 binary installed on your system, or it won’t run! (This is a temporary limitation.)

Now there is a bug today where the installer won’t start the service for you. You’ll need to do:

    sudo /etc/init.d/vmware-wsx-server start

After that, you should be able to point your browser to http://localhost:8888/. If you want to reach your VM from outside your network, just port forward this one port in your router and you’ll be set.

You’ll log in with your system’s username and password. No need to create a new account.

Windows is another story. We don’t have a build out just yet, but stay tuned on this. I’ll make an announcement when that’s ready.

What’s the plan going forward?

I can’t speak to our long-term plans, but as I continue to work on the WSX tech previews, my main goals are to make it faster, improve browser compatibility, and make it easier to interact with your VM. This means gestures (two-finger scrolling in place of a scroll wheel on tablets), gamepad controls (play Portal 2 on your iPad!), multitouch, and whatever else we can figure out.

These are my goals, and not necessarily those of VMware’s, so don’t hold the company to anything I say here!

Limitations and Bugs

There are some known limitations and bugs in this build of WSX. Please remember, this is a new prototype, and is not a finalized product!

  • You must have the python2.6 binary installed on Linux for this to run.
  • On the iPad, the on-screen keyboard is currently a little broken when it comes to capital letters and punctuation (anything involving Shift, basically).
  • We use the on-screen keyboards on mobile devices, which don’t contain things like Control keys, function keys, etc. So for now, those keys aren’t available. Looking into proper solutions here.
  • There’s no sound. Sorry if I confused some of you! There’s some things we’re waiting for in modern browsers before we implement this. No ETA or promises from me.
  • You can run against Workstation 8 Shared VMs (I think?), but you won’t see as good performance. Same with ESXi VMs.

Feedback?

If you try playing around with WSX, I’d love to hear about it. This has been a pet project of mine for the past few months, and I’m pretty excited about it. Yes, there are rough edges that we know about and will be smoothing out as we go forward, but I think it makes for a great prototype, and certainly one I’m starting to love using.

A Proud Moment: VMware Workstation 8

Today is kind of a career highlight for me. A moment I’m especially proud of. We just released VMware Workstation 8. Code-named “Nitrogen,” this release has been in the planning stages since around the time I joined VMware 7 years ago. It has been in active development for the past 3 years. Easily the longest development cycle we’ve had for Workstation, but also easily the best release we’ve ever done.

Previous users of Workstation will notice quite a lot of improvements to this release. We have a lot of changes, but I want to go into a few that I’ve worked on over the past three years, which I think are of particular interest.

Remoting

Remote Server Connection

This is the big one.

Workstation 8 can share VMs with other Workstation 8 clients. You can run a VM on one system (say, a beefy desktop machine in the back room) and access them from another (say, a light-weight laptop). All the processing happens on the machine running the VM. They can be made to start up along with the system, so you don’t even need Workstation running. You don’t even need X (on Linux).

Users of VMware Server or GSX should find this familiar. We’ve essentially succeeded the Server product with this release, with more features than Server ever had. For instance, one client can connect to multiple servers at once, alongside all your existing VMs.

That’s not all, though. You can also connect to ESXi/vSphere. As a developer, this is something I take advantage of nearly every day. I have an ESXi box running in my back room with several VMs for testing, and a couple for in-home servers. By running on ESXi, I minimize the overhead of a standard operating system, and gain a bunch of management capabilities, but previously I had to use vSphere Client to connect to it. Now I can just talk to it with Workstation.

Hear that, Linux admins? You don’t need vSphere Client running on Windows to connect to your ESXi/vSphere box anymore. That’s a big deal. (Unless you need to do some more advanced management tasks — we’re more about using the VMs, and light customization).

VM Uploading

We also make it easy to upload VMs to an ESXi/vSphere box. Connect to another server, drag a local VM onto it, and the VM will convert and upload directly to it. Super easy. Developing a VM locally and putting it up on a server as needed is just a simple drag-and-drop operation now.

No More Teams

Thumbnail Bar

Teams was a feature that we’ve wanted to rework for a long time. For those who aren’t familiar with them, Teams was a way to group several related VMs together (say, parts of a test server deployment) such that they could be viewed at the same time with a little live thumbnail bar. It offered some support for private virtual networks between them, with each NIC being able to simulate packet loss and different bandwidth limits.

We felt that these features shouldn’t have been made specific to “special” VMs like they were, so we tore the whole thing apart while preserving all the features.

Now, every VM’s NIC can simulate packet loss and bandwidth limits. Any VMs already together in some folder or other part of the inventory can be viewed together with live thumbnails, just like Teams. Any VM on the local system can be part of any other VM’s private virtual network.

It’s much more flexible. The restrictions are gone, and we’re back to using standard VMs, not special “Team VMs.”

Inventory Improvements

Inventory Filtering

You may have noticed the search field in the inventory in my screenshots. You can now filter the listed VMs by different criteria. Show the powered on VMs, the favorites, or search for VMs. Searching will take into account their name, guest OS, or data in the Description field in the VM. The Description searching is particularly helpful, if you’re good at documenting/listing what’s in a VM that you may care about (IE6, for instance).

Favorites

Favorites was reworked. It used to be that every VM in the sidebar was a “favorite.” Now we list the actual local VMs, and we don’t call them favorites. Instead, you can mark one of the listed VMs as a favorite (by clicking a little star beside it) and filter on that.

UI Improvements

Folder Thumbnail View

We’ve streamlined the UI quite a bit. All our menus are smaller and better organized. Our summary pages are cleaner and highlights the major things you want to see.

We have new ways of navigating your VMs, which is especially handy on large servers. You now get a tab for any folder-like node in the inventory showing your VMs in either a list view (with info on power states) or a zoomable live thumbnail view showing what’s happening on each VM.

And Much More

That’s just a few of the major things. There’s many, many more things in this release, but the official release notes will cover that better than me. (Honestly, I’ve been developing and using this release for so long, it’s hard to even remember what was added!)

Tip of the Hat

A lot of great people worked on this release. The engineers that developed the various components across the company. The QA groups who have provided valuable testing to make sure this was a solid release. The product marketing and management teams who kept us going and help draft the goals of this release and market it. The doc writers who spent countless hours documenting all the things we’ve done. Upper management who allowed us to take a risk with this version. Our beta testers who went through and gave us good feedback and sanity checks. And many others who I’m sure I’m forgetting.

I said this already, but I’m so proud of this release and what we’ve accomplished. More effort went into this than you would believe, and I really think it shows.

And now that we’re done, we’re on to brainstorming the next few years of Workstation.

ThinkPad T520/W520, NVIDIA, Wide Gamuts, and You!

Update: Linking to a better color profile below.

Update 2: I narrowed down the ACPI issues, and have a better solution.

I recently purchased a brand new, maxed out ThinkPad T520. My old laptop was 4 years old, and while I absolutely loved it (and still do!), an opportunity came up to get a T520 with a nice discount. So I took advantage of that. I like large screen resolutions, as I tend to have a few thousand windows on screen at any given point in time, so I went with the 1920×1080 option, and put Ubuntu Linux on it.

And let me tell you, for an Ubuntu-certified laptop, it sure didn’t work out of the box.

Optimus!

It took a lot of effort to get Ubuntu working, due to some hardware problems. The biggest issue was the display adapter. The T520/W520 (and I believe the T420, etc.) come with two graphics chipsets: An Intel something-or-other, and an NVidia 4200M. By default, this is in “Optimus” mode, meaning that the OS can essentially switch between the cards for performance/power savings reasons, depending on use.

Not surprisingly, this does not work on Linux.

Your system will try to use the Intel card. You won’t have any 3D, and try as you might, that NVIDIA card just will not work. It’s maddening, but there’s a solution. One with its own set of problems, but at least it gets you there..

The trick, it turns out, is to go into the BIOS, go into the video settings, and switch to “Discrete Graphics” and disable auto-detection of Optimus. Once you do this, your NVIDIA card will work! You’ll get 3D, and it’ll be fast and smooth and so wonderful.

If you can boot, that is.

Once I switched over, I found I could no longer boot. Now this was 4:30 in the morning and my brain stopped functioning, so I wasn’t making all the connections. All I know is that booting locked up, and when I went into recovery mode, I started seeing I/O errors on my brand new 160GB SSD. Figuring it was just my luck, I decided I’d call Lenovo in the morning and get a new one. If this sounds at all familiar, stay calm! It’s not your SSD, and your system is not hosed. It’s ACPI.

(Of course it’s ACPI… Nobody ever said it was the Year of the Linux Laptop.*)

Update: I previously said that passing acpi=off in grub would fix things. That disabled battery and other stuff, though. The new solution below is far better. Suspend/resume and brightness work!

So, to fix that, go back to Intel graphics, edit your /etc/default/grub file, and set GRUB_CMDLINE_LINUX_DEFAULT to:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash acpi=noirq"

Unfortunately, you won’t get Suspend/Resume, and I think the battery monitor is busted, but you’ll get Hibernate. It’ll mostly work. Sometimes. It’s Linux, afterall.

I’m not sure if this is needed anymore, but you may also need to add this to the “Device” section in your xorg.conf:

Option "RegistryDwords" "EnableBrightnessControl=1"

Once you have a working X/Unity/Compiz/3D/Minecraft setup working, you may notice some problems with colors….

Wide Gamut

The T520/W520/T420/etc. line of ThinkPads have a new 95% Wide Gamut LED display. It makes everything all bright and nice. And it butchers your colors.

Applications that support color management should in theory compensate for this. Not all apps do, though. What bothered me was my web browsing experience. Every color was broken, and as someone who writes webapps, I needed accurate colors. I was about to throw out this laptop before I managed to figure out a solution.

Modern Ubuntus should come with a Color Profiles control panel under System -> Preferences. From here, you can load ICM profiles and get things working. That will solve some of your issues, so let’s start there. Note that this is in Ubuntu 11.04 (I think), but I’m running 10.10 (I thought originally 11.04 was the source of my problems), so for me I had to:

apt-get install gnome-color-manager

And here, I the Color Profiles applet was causing problems, and I couldn’t directly use it, so I installed dispcalGUI, which allowed me to load in a profile and install it into Color Profiles, and activate it.

On some site, I found a working color profile (ICM file). I don’t know where I originally found it, but you can download it here. Note that this is only tested on my 1920×1080 display, so YMMV.

I’m not confident that this profile is 100% correct, but it’s close. At least as far as I can tell.

You’re free to try that profile, but I found a better one which preserves the crispness of the display with the actual accuracy you’d want on the web. WordPress is even looking correct now.

NotebookCheck’s review of the ThinkPad W520 links to a color profile that works much better.

Firefox Color Profiles

Now, you’ll still notice problems with Firefox and Google Chrome. Chrome doesn’t seem to understand color profiles, but Firefox does. You just have to tweak it.

In Firefox, go to about:config. In there, search for gfx.color_management. Set gfx.color_management.mode to 1, and then set gfx.color_management.display_profile to the path of that ThinkPad ICM file I linked to earlier.

Restart Firefox. You should now see more or less correct colors!

Now it’s not perfect. The ThinkPad screen is very bright, and some things can get a bit washed out and perhaps slightly tinted. As I type this, I’m noticing that the WordPress UI is a bit off, and things blend together more than they should (lots of light grays on a bright screen). But it should be much better than it was!

If you use a T520/W520/T420/etc. and have other solutions or tips for color management on Linux, I’d love to hear about them! And hopefully this saved someone else hours or days of rage.

* Thanks to James Farwell for the “Year of the Linux Laptop” snark.

I Invented Port Knocking

Let me tell you about something that’s been bothering me for a while.

I invented Port Knocking. No, really. In 2002.

According to portknocking.org, it was invented by Martin Krzywinski in 2003. I’m not here to debate that he didn’t come up with the idea separately, and choose the same names (it’s a pretty good name for the technology). But I do want to make it clear, for the record.

Wait, hold on, what’s Port Knocking?

Oh, got ahead of myself there.

Port Knocking is a security method where you can cloak a network completely (close all ports or put them in stealth mode) and yet still allow access from any computer in the world, by way of a sequence of “knocks” on a predefined list of ports.

The server can specify a list of ports (say, 53, 91, 2005, 2131, 7) and monitor to see if there are attempts to open them. If an outside computer accesses each of these ports in sequence, without hitting any other ports, and within a time period, the server can open a select set of ports (separate from the knock list) to that IP address only.

In my original designs, before opening the ports after a successful knock sequence, an authentication port would be opened at a predefined port, which the client would have to access, exchanging credentials, before the ports would be open.

And why the controversy?

First, some history.

In mid-2002, I was 18 and interested in security, amongst other things. Along with writing code for Pidgin (then Gaim), and a couple other projects, I was fooling around with firewalls and such.

I had this idea one morning while in the shower to add another layer of security. I really wanted to be able to completely close off my network, but still access it when out of town. I can’t tell you how it came into my head. Just a moment of inspiration. I wasn’t even really looking for another project, just brainstorming, but I liked the idea too much. I started writing code and made it work.

It was a while before I discussed it publicly on my old blog on Advogato. There are many posts, but I’ll highlight a couple here, where I introduce what I was working on:

The blog is full of lots of old teenage angst, so ignore most of it, but I spend the next few weeks going over my progress, answering questions from people who are asking for more information, etc. I was very open about it.

At one point a couple months later, I realized this was stupid. I had a good idea. I should patent it. I took it down for a while. This was after I had already put up the sourcecode, though, and many people had it.

Now, in retrospect, I should have made this into a full-on open source project and gained the recognition myself, continued development. But I was too busy with other things and didn’t really want another major product on my hands. I remember at one point I thought, “maybe I can sell this to a security company, or patent it!”

And since then…

One day, I opened a magazine and saw “port knocking” on the cover. My heart skipped a beat. Somebody wrote an article on my port knocking! I opened the magazine and read through it. “Invented by… Michael Krzywinski? What?!” I re-read to make sure. It was all my terminology, my methods. I was floored.

By that point, he made a name for himself as the inventor. And again, I’m not trying to discredit him, because he very well may have come up with the same thing separately. But it stung, because I had a great idea, a year before he wrote a paper on it, and I didn’t promote it the way he did.

Lesson learned

This is one of those life lessons. You always regret what happened, but you use it to make better decisions in the future. These days, I’m happy working on some awesome products. My day job at VMware and my highly successful code review software, Review Board (for which we’ve recently started a company).

Now, if I have a good idea, I make sure it’s heard, and demonstrated, far and wide. Truly great ideas don’t really come that often, so when you have one, make sure you do something with it, or you may end up regretting it for years to come.

Sentience discovered in the Linux kernel

Ladies and gentlemen, after much experimentation, I have made a remarkable discovery. Perhaps the very first case of a sentient AI has been discovered, sitting right under our noses, in the Linux Kernel. With such a complicated codebase that has evolved greatly over the years, there are certainly more surprising places for it to spring up, but it’s still quite unexpected.

And where, specifically, has this sentience manifested itself? The suspend/resume code.

See now, like many of you, I’ve dealt with the instabilities of suspend/resume. I’ve considered it to just be buggy, unreliable, and possibly incompatible with my hardware. That is, until I realized that there’s a pattern. One that began to make a sort of sense.

A couple months back, I gave suspend/resume another shot, and to my surprise it worked. I figured that Ubuntu 10.04 finally fixed it, but it still wasn’t perfect. I still noticed problems.

The first thing I noticed was that when I unsuspended at work, I couldn’t use my volume keys. Everything else was fine, but my laptop’s volume keys didn’t register as a key press on anything. If I suspended again and brought it back home, the keys would work fine. If I suspended at home and resumed at home, I wouldn’t have the volume key problem. Weird, but just buggy, right?

It was a couple weekends ago when I suspended my laptop to take it somewhere. It wouldn’t suspend at all. Just hard-locked. This continued until the week, when it worked again. Last weekend? Same problem, couldn’t suspend. Monday, it worked fine.

It was then that I realized suspend/resume was breaking deliberately! See, my laptop feels more comfortable at home, less so at work but it tolerates it (with some complaining), but absolutely doesn’t want to leave during the weekend. It’s like a cat that just wants to be in a familiar environment, selfishly vying for your attention through mischievous acts. Look at it hard enough and the pattern emerges. It’s undeniable.

That got me thinking. What other possible instances of AI have we been misconstruing as bugs or random glitches? All those inter-connected street lights that occasionally shut off as you walk underneath them? Maybe they’re just shy, or they hate you. Maybe NES cartridges just found being blown stimulating.

So remember guys. Windows suspend/resume may work just fine. Mac too. But Linux’s suspend/resume isn’t a buggy pile of crap. It’s an intelligent buggy pile of crap, that just wants to be loved.

Looking Back on Review Board

Just over 3.5 years ago, David Trowbridge and I spent some time discussing the annoyances of the typical patch submission and code review processes in the open source projects we participated in and at companies, and decided to play with some ideas for improving this. At the time, we knew very little about what we intended to do. We had a name for it pretty early on, but that was about all we had. We didn’t even know whether we’d get past an early prototyping stage. But here it is, over 3 years later, and we have the leading open source code review tool with an active support and development community, hundreds of companies using it, and exciting new innovations for aiding in the code review process.

I was thinking a few days ago about how far we’ve come and some of the decisions we made along the way. I went digging through our commit history in order to relive some of the past of our little project. Since so few people were even aware of Review Board’s existence at the time, I thought I’d share some of our history with you. Particularly the interesting and funny bits.

“Add the reviewboard.”

Commit #1. The very first thing we put in our Subversion tree on September 27, 2006. I don’t even remember what was in this change now. We transitioned to Git last year and this commit is now just plain empty. Maybe it was jut the directory structure? Who can say.

Early on, we didn’t refer to “Review Board” as a proper name. It was generally “the reviewboard” or something similar. The codebase was young. We didn’t actually do code review on the project at this point (and it shows!). The first few months are littered with odd or nonsensical commit messages, small breakages, and bad decisions.

A few of my favorite commit messages are:

  • “I suck. Make submitting of reviews.”
  • “Don’t stuff the list of files in the bug list. It’s impolite.”
  • “Avoid failing out with Christian’s wacko form”
  • “Gum.”
  • “Holy apple pancakes. It worked!”
  • “I suck… The array was empty… The tests never had a chance to fail. :(“
  • “‘This is a summary’ sucks. Now we use fortune for the summary, description, and testing done. ‘You’re ugly and your mother dresses you funny.'”
  • “Unbreak things before ChipX86 notices”
  • “I’m just… garhgh”

Nowadays, our commit messages look nothing like that, but that’s the fun of a new project. You get to go commit-crazy while you try to figure out what you’re building.

Dashboard, quips and fortunes

The UI of old looked quite different than the UI of today.

We had a dashboard from the very beginning (before the review request pages, even) but it wasn’t anything like the dashboard we had today. It was a simple page with a table containing all outgoing review requests and a table containing incoming review requests. But it also had one more thing: quips.

The beginning of quips functionality was being built. Quips are just little random quotes that are inserted in the UI. I think the plan was to put quips on certain pages, making Review Board a little more fun. We were using them in the dashboard for empty lists, with variations all saying something about the dashboard being empty. Quips are a neat feature that just never survived the early days of development.

Fortunes are similar. On Linux/Unix systems, there’s a little program called “fortune” that just displays a random quote. Since we at first had to test review request functionality without actually having a repository backend of any sort, and we didn’t want to input all the information each time, we just used fortune to generate the summary, description and testing done text. This made for some really funny review requests early on, but this is of course something that had no reason to survive initial development.

Sometimes we would create a bunch of review requests just to see what kind of quotes we’d get. 🙂

Multiple repositories? Almost didn’t happen.

One of the really critical parts of Review Board today is the ability to talk to a variety of different types of repositories in one instance. But, it turns out, this almost didn’t happen.

The initial goals were not that ambitious. Review Board talked to one repository per instance. Everything was basically hard-coded with one repository in mind. That type of repository, as well as its information, was customizable. You just couldn’t have more than one. At the time, this wasn’t a problem, but it didn’t take long until we had a need to talk to two repositories.

We discussed this and at first decided that if we needed to talk to two repositories, we could just set up two instances. It would have been a lot of work to update it for multiple repositories, after all. And really, this was a small project. Who would really need more than a couple repositories? This started to nag at me, though, and so I spent a couple nights rewriting all of the code as an experiment. It ended up working pretty nicely, and we were able to ditch the multi-instance model.

The importance of rewards

It’s always nice to have a little reward for milestones. Developers sometimes compete over cool bug numbers, revisions, etc. Initially, we were going to use quips to add some fun to the site, but we ended up settling on our current trophy system.

One of our first Review Board instances started to approach review request #1000, which was a huge milestone for us. I decided to commemorate the event by staying up and quickly hacking in a hidden feature for showing a trophy for review request #1000. The way we implemented it, you’d see the first ever trophy at 1,000, and from there you’d see it at every milestone number (1,000, 2,000, 3,000, 10,000, etc.). I didn’t want to stop there, though, so I added support for a second type of trophy, one that has confused people with its appearance to this day. Mission complete.

Of course, when we updated the server and someone finally hit 1000, it triggered a bug in the new trophy code and broke his review request. Oh well, I tried.

Diff viewers are hard

If I could pick one point during the whole history of Review Board where I was ready to completely give up, it would be during the creation of our diff viewer. All three diff viewers.

See, the first diff viewer was a complete and total hack. We generated a side-by-side diff using the diff tools and just parsed the output, basically generating a table of that. It was ugly, though, and limiting. It also caused problems where text on a row would either be truncated or would break the parser. I spent a long time working on this before I totally gave up and went on to try a new approach.

My second approach was closer to what we have today, but also limiting and very, very buggy. We were using Python’s built-in diff generation module, which implements a basic diff algorithm. It gave us insert and delete information, but not replace information. We had to hack that in ourselves, and it was really a hack. Try taking a bunch of inserts and deletes and find out which of those are really changed lines. No, really, try it. It’s harder than you think, and it’ll often be wrong.

Still, we stuck with this for a long time. It was slow, buggy, and didn’t generate the sort of output people expected from diff tools. Most people see diffs from GNU Diff, which implements the Meyers Diff Algorithm (with a few additions and tweaks). These Meyers diffs are much nicer to view than what Python gave us. Another problem we hit was that we didn’t have real line number information, so we had to output fake line numbers. They weren’t really line numbers so much as row numbers in the table. Ugh. Even getting this far was really hard and frustrating, and the result still wasn’t good.

Attempt #3. I decided to build our own diff parser and generator from scratch. What a project. I knew nothing about diff generation and hardly knew where to start. I spent probably a good month or so just trying to work on this new diff code, and was so close to giving up so many times. It ended up being completely worth it, though, as we ended up with a very nice, extensible diff parser.

Without that third attempt, we’d be in the stone age. Review Board would not be as nice to use. We wouldn’t have inter-line diffs (where we highlight what changed in a replace line), syntax highlighting, move detection (coming in 1.5), or function/class headers (where we show which function/class the part of the diff is in — also coming in 1.5).

What else…

Well, there’s a lot more I could talk about. Our initial attempts at JavaScript code for the UI, our trials and challenges with database migration, or our early problems storing diffs with different encodings in databases. This is getting long, though, so I’ll cover these in another post on lessons learned.

Goodbye, my friend

Goodbye, Vinay

I lost a good friend Sunday.

Vinay Venkatesh, also known as djgraphite, was a friend and co-worker at VMware. We had known each other for many years, since before he joined VMware, from the #adium and #growl channels on irc.freenode.net back when I worked on libgaim and he worked on Growl. Vinay was always helpful , friendly, passionate, and full of creative ideas. This extended to his work on VMware Fusion

A few years ago, Vinay interviewed for a job at VMware in my team. We hired him for the relatively new Fusion product for the Mac. This was my first in-person experience with him, and we quickly became friends. I remember spending hours in his office talking about all sorts of things. New games coming out that we wanted to play, projects we were working on, new gadgets, ideas for Review Board, architectural changes we wanted to make to our products at work, what we did on the weekends… Anything and everything, really.

While we developed separate projects at work, we often ran ideas past each other. Vinay shared my desire to improve the common and per-platform code bases we each worked on, and while we didn’t always find the time to implement each idea, much of my discussions with him led to improvements in all of our desktop products: Workstation, Player and Fusion. Over the past month, I’ve spent considerable time on a project that was largely his brain-child. The details aren’t important, but suffice to say that it’s an important part of the future versions of all our desktop products. Every step of the way, I consulted with him, making sure I was on the right track, asking for advice in the design, and getting code reviews. Continuing on that project without him by my side is something I’m certainly not looking forward to.

His work was just a small part of his life, though. Most important to him was his friends and his family. Making friends with Vinay was easy. He was inviting, outgoing, funny, and loved meeting new people. He had a lot of friends at work and outside of work. I thought I knew a good number of them, but I realized since just how few I knew. We were important to him and he let us know that.

It was also no secret to anybody who knew him just how close he was to his family. He spoke of them often, with praise and love. He would tell us about his sister, how happy he was that she was getting married, how excited he was that she was moving closer to him. He would talk about his parents and tell us how every time he visited them they would ask when he was getting married. We would joke that one day he’d return after a trip with a bride around his arm.

When I think of Vinay, and this is how I always pictured him, I think of him laughing. He was generally a very happy, upbeat guy. Liked to joke around, share stories, and spend time with friends. One of the things he really helped drive at VMware within our team was our Thursday Movie Night. Every Thursday (more or less) we go to dinner and then come back to the office and watch a movie. Vinay loved Movie Nights with us and often provided the movies and dinner recommendations. Outside of work he’d host parties at the house he shared with many of our friends. His last party, which I regretfully didn’t attend due to conflicting plans, was a Halloween party on Friday the 30th. I hear it was a lot of fun, and I’m glad he was able to enjoy himself one last time.

On Sunday, around 1PM PST, I got the terrible news. Vinay had been in a motorcycle accident, and died on the operating table.

I got the news on Twitter, shortly after dropping off a mutual friend at the train station. I didn’t believe it at first. My mind said “No, this is a joke or just a misunderstanding,” but part of me knew the truth. I quickly dialed people, trying to find out what happened. I reached my friend Scott at the hospital, who was with Vinay when it happened. He told me the news that broke me.

I wanted to blame someone, but this was one of those freak accidents. He was with a group of people, riding his motorcycle, when he hit a groove in the road that knocked him off his bike. There were no external injuries, but they couldn’t stop the internal bleeding. He died shortly after.

News spread vast, over both Twitter and Facebook. A group of us organized at the house he shared with others, trying to comfort each other and come to terms with what had happened. None of us wanted to believe it, but we couldn’t deny it had happened. It was a night of hell. The next day wasn’t any better. Very few of us even attempted to go into work, and those that did gave up being productive quickly. Throughout the day, information spread, again over Twitter and Facebook, about the funeral plans, which were set for Tuesday the 3rd.

The funeral was hard, but it was a nice ceremony, as nice as these things go anyway. It was evident just how many people cared for Vinay and how far his influence had spread. The room we were in was not small, but it was so packed that people were overflowing into a second room. The turnout was huge. After we paid our final respects, many of us went back to the house, comforted each other, and shared stories.

It was a tragedy, and certainly too soon. I do find some comfort in knowing that Vinay went out doing what he loved to do. It also brought people together. I met some great people from one of his many groups of friends tonight, as well as finally meeting his family. I wish these meetings would have happened in better circumstances, but I’m certain Vinay would be happy to know that in some way, he brought his friends closer.

Rest in peace, my friend. We love you, and we’ll never forget you.

Review Board 1.0 Released!

Review Board 1.0

Tonight, we hit a milestone in the Review Board project that we’ve been working toward for over two years. We finally pushed out our 1.0 release. A lot of blood, sweat and tears went into this release (okay, so not literally, but it was A LOT OF WORK!). The last few months in particular have been challenging, as we’ve had to solve some tricky bugs and scalability problems, but the end result is pretty great.

Just a short while ago, we announced the release and put up an overview of the entire release and product. We’ve already had some nice congratulatory e-mails and tweets, which is really nice 🙂

Some stats for this release:

  • 2 years, 9 months, 25 days have passed since our first commit.
  • 120 contributors have contributed to Review Board so far (in terms of code contributions).
  • 2,019 commits were made.
  • 899 review requests have been posted to our project’s actual Review Board server. 1,650 users are registered on there.
  • Our demo server, in comparison, has 2,082 review requests filed and 10,154 users.
  • 938 bugs were filed. 812 were fixed.
  • 232 feature requests were filed. 101 were implemented. Most remaining ones are scheduled for releases.
  • An estimated 200+ companies are now using Review Board. 26 have let us list them publicly.
  • The largest known Review Board install has over 83,000 filed review requests and over 2,000 users, doing upwards of 10GB of traffic per day.
  • 5 presentations on Review Board are known to have been given, 3 by us, 2 by others.
  • 552 users have joined our main mailing list, and 3,674 e-mails have been sent.

Now that Review Board 1.0 is out, we can get started on some awesome new features we’ve had planned. I have a little notebook full of ideas for our 1.1 and 1.5 releases (which may become 1.5 and 2.0, respectively, as this list grows). Some of the new features are actually ready to be committed within the next couple of days, so those of you using nightlies will start to see them soon.

We were accepted into this year’s Summer of Code, and have three students working on exciting projects for us, so hopefully we’ll start to see these trickle into the upcoming nightlies as well. Among these projects include diff viewer improvements (moved region detection, better whitespace-only change detection), IDE integration with Eclipse, and improved notification hooks and e-mail support.

We’re also working on providing support for third-party extensions, which will allow developers to extend Review Board in new, exciting ways without having to modify Review Board itself. This is especially handy for companies who wish to integrate better with their sandboxes, bug trackers or unit testing services. This will likely land in 1.5 (2.0?) at the earliest, as it’s a large change, but the code for this mostly works today. It’s just a matter of getting the codebase ready and figuring out what APIs we want to stabilize and expose.

As I mentioned in the release announcement, we’re planning a release party, tentatively on July 11th, 2009, in the Bay Area (somewhere around Palo Alto, CA). If any Review Board users want to join us, please RSVP!

Review Board: Summer of Code, Roadmap and Future Plans

Summer of Code

This year, we (the Review Board project) was given the opportunity to participate in Google’s Summer of Code. We’ve received some great student proposals so far, and I think we’ll see exciting work done on Review Board this summer.

The deadline for Summer of Code is coming up fast (April 3rd, 19:00 UTC). If you’re interested in working on Review Board and haven’t yet applied, it’s not too late, but you’ll want to hurry. Skim through our ideas page and, if you find something interesting or have a great idea not listed here, then apply and tell us your plans. I can say we’ve received several proposals so far for the installer and admin UI, so unless you feel strongly about either of those, you’ll increase your chances with other proposals.

We’re also offering free Review Board hosting for open source projects participating in Summer of Code. If you’re a mentoring organization and would like to give Review Board a try for reviewing and managing student code, go ahead and contact us and we’ll get you set up.

Roadmap

We’re finally nearing 1.0. We recently put out our 1.0 beta 2 release and are now in a feature freeze. We’re working to get some bug, performance and usability fixes in for beta 3, which I’m shooting for in a few weeks. Then we’ll branch for 1.0, put out a Release Candidate or two, and then finally release 1.0!

There’s a lot of really cool features planned after 1.0, namely extensions and policy customization.

Extensions

Our bug tracker is filled with feature requests for all kinds of things, ranging from bug tracker integration, instant messaging, a method for offering bribes for code reviews, and so on. We clearly can’t put all the requested features in the codebase, so we’ve decided instead to add support for third-party extensions. Coming soon, developers will be able to write extensions to Review Board in the form of Python modules to extend or alter the functionality of Review Board. The extension framework will allow them to do the following:

  • Access the database using the existing Review Board database models.
  • Add new database models for storing data.
  • Listen for signals (new review request published, review request submitted, etc.) and act on them.
  • Add custom URLs.
  • Replace existing URLs, for advanced capabilities such as replacing the diff viewer.
  • Add new API handlers.
  • Add “action” links to existing review requests and reviews.
  • Add columns and sidebar entries to the dashboard.
  • Add pages to the administration UI.
  • Communicate with other extensions.
  • Provide a settings page, which stores data in Review Board-provided models (we even auto-generate the settings page for the extension by default).
  • And more!

A lot of this already exists in a private development branch, and it will be one of our primary focuses as soon as 1.0 goes out.

In time, we’ll add a new section to the Review Board website where developers can list their extensions for download and for sale. Administrators will be able to browse and search for extensions directly from the administration UI and install them without having to even open a terminal (in most cases).

We’re hoping this will solve a lot of in-house integration issues. For example, many companies have custom sandbox architectures, bug trackers, and statistics software which they’ll now be able to tie in with Review Board.

Policy Customization

We’ve found that a lot of companies have very specific ways they want to handle policy and access restrictions. For example, many companies want to limit who can see certain parts of a repository (and therefore certain diffs), or want to allow anybody to create review groups, or want to disallow people from joining review groups. Some also want to dictate what constitutes approval for submitting a change.

We’re looking into the various requests and attempting to come up with a policy model that is flexible enough to handle these needs. One of the ideas is to provide some basic level of access control on a per-repository, per-path, and per-group basis. We’d then piggy-back on the extension framework to allow for more specific policy control. The advantage is that developers could write their own policy rules that interface with some part of their company’s infrastructure.

If people have any input on this, we’d love to hear it.

Improving browser performance in Review Board

This past Sunday, I landed a set of changes into Review Board that provide improved performance, such as aggressive browser-side caching of media and pages. It’s just a start, but has already significantly reduced page load times in all of my tests, in some case by several seconds. We implemented these methods for Review Board, but they’re methods that can be applied to any Django project out there.

There are several key things that Review Board now does to improve performance:

  • Tells browsers to cache all media for one year.
  • Only sends page data if new data is available.
  • Compresses all media files to reduce transfer time.
  • Parallelizes media downloads.
  • Loads CSS files as early as possible.
  • Loads JavaScript files as late as possible.
  • Progressively loads expensive data.

A lot of the performance improvements come straight from Yahoo!’s Best Practices for Speeding Up Your Site. We’re not doing everything there yet, but we’re working toward it. That’s a great resource, by the way, and I recommend that everyone who has even made a website before go and read it.

So what do the above techniques buy us, and how are we doing them? Let me go into more details…

Caching all media for a year

The average site has one or more CSS files, JavaScript files, and several images. This translates to a lot of requests to the server, which may leave the site feeling slow. On top of this, a browser only makes a few requests to a server at a time, in order to avoid swamping the server, which will further hinder load times. This happens every time a user visits a page on your site.

Aggressive caching makes a huge difference and can greatly reduce load times for users. Review Board now tells the browser to cache media files for a year. Once a user downloads a JavaScript or CSS file, they won’t have to download it again, meaning that in general the only requests the browser needs to make is for the page requests and AJAX requests.

The big pitfall with long-term caching is that the cached resources can go stale. For example, if a new version of an image was uploaded, the browser wouldn’t even know about it, since it was told it should keep its old version for a year before checking again.

We solve this by introducing “media serials,” timestamps that are appended to all media paths. Instead of caching /js/myscript.js, the browser would cache /js/myscript.js?1273618736.

These media serials are computed on the first page request by our djblets.util.context_processors.ajaxSerial context processor. This quickly scans all media files known to the program, finding out the latest modification timestamp. It then provides a {{MEDIA_SERIAL}} variable for templates to append to media URLs as part of the query string.

The benefit to this method is that we can cache media files for a year and not worry about users having stale cached resources the next time we upgrade a copy of Review Board. The filenames requested will be different, browsers will see that the new file is not in the cache, and make a request, caching the new file for a year.

Only send page data if new data is available

Aggressive caching of media files is great and saves a lot of time, but it doesn’t help for dynamically generated content. For this, we need a new strategy.

When a browser makes a request, it can send a If-Modified-Since header to the server containing the Last-Modified value it received the last time it downloaded that page. This is a very valuable header, and there’s some things we can do with it to save both the server and the browser a lot of trouble.

If the browser sends If-Modified-Since, and we know that no new data has been generated since the timestamp provided, we can send an HttpResponseNotModified (HTTP response code 304). This will tell the browser it already has the newest version of the page. The sooner we do this, the better, as it means we don’t have to waste time building templates or doing any expensive database queries.

Djblets, once again, provides some functions to help out here: djblets.util.http.set_last_modified and djblets.util.http.get_modified_since.

The general usage pattern is that we first build a timestamp representing the latest version of the page. This could be the timestamp for a particular object that the page represents. We then check if we can bail early by calling:

if get_modified_since(request, timestamp):
    return HttpResponseNotModified()

Further down, after building the page, we must set the Last-Modified timestamp, using the same timestamp as above, like so:

set_last_modified(response, timestamp)

We’re using this in only a few places right now, such as the review request details page, but it drastically improves load times. If the review request hasn’t changed and nobody’s commented on it since the browser last loaded the page, a reload of the page will be almost instant.

Compress all media files

Our Apache and lighttpd config files now enable compression by default. By compressing these files, we can turn a relatively large JavaScript file (such as the jquery and jquery-ui files) into a very small file before sending it over to the browser. This reduces transfer times at the expense of compression/decompression time (which is small enough to not worry for deployments of this size, and can be offset by caching of compressed files server-side).

Parallelize media downloads

It’s important to not mix loads of media files of different types. The browser parallelizes media downloads of the same type, in page load order, but if you load one CSS file, one JavaScript file, another CSS file, and then another JavaScript file, the browser will only attempt one load at a time. If you load all the CSS files before all JavaScript files, it will parallelize the CSS file download and then the JavaScript downloads. By enforcing the separation of loads, we can achieve faster page download/render times.

Load CSS files as soon as possible

Loading CSS files before the browser starts to display the page can make the page appear to load smoother. The browser will already know how things should look and will lay the page out accordingly, instead of laying the page out once and then updating that once the CSS files have loaded.

Load JavaScript files as late as possible

JavaScript loads block the browser, as the browser must parse and interpet the JavaScript before it can continue. Sometimes it’s necessary to load a JavaScript file early, but in many cases the files can be loaded late. When possible, we load JavaScript files at the very end of the document body so that they won’t even begin downloading until the page has rendered. This provides noticeable performance for script-heavy pages.

Progressively load expensive data

There are types of data that are just too expensive to load along with the rest of the page. For a long time, Review Board would parse and render fragments of a diff for display in the review request page, but that meant that before the page could load, Review Board would need to do the following:

  1. Query the list of all comments.
  2. Fetch every file commented on.
  3. Apply the stored patch to each file.
  4. Diff between the original and patched files.
  5. Render the portion of the diff commented on into the page.

This became very time-consuming, and if a server was down, the page wasn’t available until everything timed out. The solution to this was to lazily load each of these diff fragments in order.

We now display a placeholder table for each diff fragment in roughly the same size of the rendered fragment (to avoid excessive page scrolling on loads). The table contains a spinner showing that something is happening, and, one-by-one (to avoid dogpiling) we load each diff fragment.

The code to render the diff fragment, by the way, takes advantage of the If-Modified-Since header and is also cached for a year. We use an AJAX_SERIAL (same principal as the MEDIA_SERIAL above) to allow for changes in new deployments.

With these caching mechanisms in place, the review request page now loads in roughly a second in many cases (or less once cached), with diff fragments coming in lazily (and then almost immediately on future loads).

More to come…

This was a great first step, but there’s more we can do. Before we hit our 1.0 release, we’re going to batch together all our CSS files and JavaScript files into a couple of combined files and then “minify” them (basically compressing them in such a way to allow for smaller files and faster load times of the interpreted data).

Again, these are techniques we’re now making use of in Review Board, but they’re not in any way specific to Review Board. Anyone out there developing websites or web applications should seriously look into ways to improve performance. I hope this was a good starting point, but seriously, do read Yahoo!’s article as well.