Memories from VMware’s Hosted UI

I wrote a tribute last week to my old team at VMware, the Hosted UI Group (aka HUG). These people were like family, and through their hard work and dedication, mutual respect and insane depth of knowledge, they built some amazing products, Workstation and Fusion being just two of them. I was so proud to be part of this team and so sad when I heard their group had been axed.

I shared my point of view on just a few of the things that made the team great. Tried to keep it short, and didn’t know who would read it, but as I write this, over 57,000 people have read my tribute, and many have shared their thoughts on the team.

Since then, our team, what we’ve been referring to as Ghosted-UI, has been out for a couple dinners, drinks, trying to figure out where everyone will end up, trying to figure out where poker’s going to be next, when the next good movies will be out, and, actually, still discussing bugs and thoughts around Workstation and Fusion. Okay, maybe we haven’t let go yet.

So here’s what I’d like to do now. I’d like to share some thoughts and pictures from a few people in our team about what made our team great, and share some comments from some of our users. Keeping it positive here.

Hosted UI Musings

Jocelyn Goldfein

Before Jocelyn left for Facebook in 2010, and then got to branch out on her own, she was the manager of Hosted UI. In fact, she’s the one who hired me back in 2004, and remained a mentor to me. My first job out of school (actually, hadn’t finished yet), and she took a chance on me, bringing me on board and helping me learn the ropes, learn to build myself up. Even paid for some driving lessons (I didn’t have a license back then).

There was a part of that that I never thought she’d have remembered, which she shared with us on Facebook:

Ok, I can add *one* thing. What Christian doesn’t mention about his aforementioned first day of work is that he anxiously showed up at 8am, in a collared shirt and slacks, nervous for his first, grown-up, corporate job.

He then got to cool his heels in the lobby for an hour waiting for the next person on his team to arrive (which was probably me around 9 or 9:30).

He immediately bonded with the team, came back the next day in jeans at a reasonable hour like 10am, and the rest, as we say, is history. 🙂

Jocelyn, with a real family of her own, shared her thoughts on what made us a family.

I 100% experienced the team as family.

To me, video games were the least of it. We were united by our shared sense of mission and care for what we were building and the community of developers and admins who used it. Our commitment to the kind of software we wanted to build and the way we would build it. Gamely celebrating each other’s lifecycle events whether that was a 21st birthday or a surprise baby shower for me which was the first event of its kind for most of the attendees. 🙂 while I haven’t stayed in close touch with everyone in the 7-8 years since I became less involved w/ the team, I’ve attended weddings, doled out career advice, helped with job hunting, new parent advice, you name it. IOW, I haven’t been there all the time, but I’ll absolutely be there when I’m needed.

For me, this team emphatically represents the possibility that you *can* form very close and lasting relationships with work colleagues, without them HAVING to also be social connections. For Christian it was both, but I don’t think I was the only one for whom it was not friendship… but still family.

When I asked for additional thoughts for this post, she brought up part of what made our products so consistently high quality, even 12 major versions in:

I feel like some of our long term rearchitecture/cleanup efforts deserve highlighting. It’s hard to have the discipline to do those. It’s not sexy or fun for the engineers, and marketing could care less b/c it doesn’t drive sales. But we pulled off some big ones b/c we had the team commitment and will to do it.

James Farwell

James (LinkedIn) joined Hosted UI in 2007, and spent most of his years since working on Fusion. He’s still with the company, just in another role, but is very much a part of our team. In a Facebook post, he shared some of his thoughts that kind of summed up our work days:

The simultaneous technical breadth and depth of this team was always stunning. You could walk past an office where 3 people were having a design discussion about how to do some complex asynchronous task while respecting the quirks of the OS X and GTK run loops and Win32 message loop. Or have a debate with someone about how best to model and manage modal dialogs in a generic fashion while still having the application “feel” like it was supposed to on each respective platform. I can’t stress that enough, so much love and care was put into having each application “feel” like a Windows app or “feel” like a Mac app despite having so much shared code. And as much ribbing as we gave each other about the other platforms, there was always so much respect.

And then you’d all go out to dinner and talk about video games.

Lee Ann Rucker

Lee Ann (LinkedIn) also joined Hosted UI in 2007, working on Fusion. I remember giving her an interview, poorly (that is, I sucked at it — I was pretty new to interviewing). She was a fantastic member of the team, really knew her stuff.

When I asked for thoughts on the team, she shared why she stayed with this product so long: Our users.

This is why I did it – because our users appreciate it. I dropped a line to the blind Fusion user [who she heard from after the news broke last week — Christian] and got this answer:

> “Hi Lee Ann. Thanks for taking the time to write, and I’m sorry to learn that you and others who have done such a good job with Fusion have been let go.
>
> It sounds like VMWare has lost a lot of institutional knowledge, and Fusion is the only accessible VM solution there is.
>
> Thanks for thinking of your blind users, many of us really appreciate all you have done.”

Jason Kasper

If you’ve used Unity on Linux or Mac, you’ve used Jason’s (LinkedIn) work. He was a remote employee, so we didn’t get to see him as often as we liked, but we chatted on IRC daily.

I did not grow up with nerd friends who were like me. I spent the first period of my life working whatever I could find in retail stores and then in corporate IT, and I definitely didn’t fit in there. But you guys… you’re all like me! Like, there’s no pretense and there’s no trying to fit in. It’s just always felt like home and family when I’ve been able to spend time with you guys.

I know it’s going to sound sad and sappy and whatever, but I just wanted to tell you guys how much you have meant to me and how much you continue to mean to me. You all have been the best 8 years of my life, personally and professionally. I love each and every one of you. And I’m going to miss being able to hang out with you in person immensely. *hug* =:)

We’ll miss it too, Jason, but are going to drag you out here kicking and screaming, one way or another.

Sujit Polpaya

Sujit (LinkedIn) was a newer member of the team, and moved to the team after I left. I’ve gotten to know him through team outings, and am glad I had that opportunity.

My tenure in the Hosted UI team is significantly less than many of you guys, but I share the pain. By far this is best team I have ever worked with – amazing people and products. I would have completed 4 years at VMware on April 30, which is just about a month after my termination date. I was looking forward to this 4-year milestone. Oh well…

Richard Bailey

Richard started off as an intern in Hosted UI, and then became a full member of the team. He left a few years back, but like many from Hosted UI, we still keep in touch on IRC, Facebook, and Twitter. He had some really nice thoughts to share with us:

The Hosted-UI Group set the gold standard by which I have judged all the teams I’ve worked with since I left (2012). We built an open environment that celebrated individuals’ strengths and supported each other’s weaknesses. We weren’t all hanging out on the weekend together (though many were) but it didn’t matter because we all cared for the product and wanted to see it succeed.

There was a dramatic breadth of technical skill (from deep kernel hacking to amazing UX intuition and user focus) and very little ego. It was an amazing place to learn post college and the best introduction to industry I could have imagined. I’m devastated to see the team disbanded but hope that the core of how that environment functioned follows each of the team as they spread out to whatever amazing things they decide to pursue.

On a more personal level I owe a lot of my adult life to the situations that arose from taking the full-time offer. Without coming to HUG I would have not met many of those who are now my closest friends, would not have started rock climbing (which was key for me to *finally* get healthy), and would not have made the connections necessary for my subsequent jobs which I have also loved. It would be an understatement to say that this team, and the internship that pulled me in, significantly altered the course of my life.

For those that were laid off you can pretty much call on me for anything and I’ll do what I can to help regardless of whether or not we overlap: that’s how family works. I’m sorry this happened but I trust you’ll be okay.

James Lin

James (LinkedIn) was one of the first people I met when I joined VMware. He’s a legend. The guy knew the codebase inside-and-out, probably better than most of us, and plowed through bugs and features like nobody’s business. He worked on the Windows side on Workstation and Player. He saw a lot of change in the company and even in the team, and I always pictured him single-handedly holding the products together until the very end, if it came down to it.

He shared his view of what made the team great and how he saw his work over his time at VMware.

I’ve been in VMware’s Hosted UI group (“HUG”; could there be a more appropriate name?) working on Workstation for almost 12 years.  I’ve seen a lot of people in HUG come and go (although I think not quite as many as in other groups), and while some of them tried to pull me away to join other companies, I never really wanted to leave.  I loved our product.  Even after nearly 12 years, I never got tired of fixing bugs; I saw each bug as a usability problem for customers.  Repetitive bugs challenged me to try to prevent future recurrences.  At VMware, there was always something for me to work on and always something new for me to learn, and it never got boring.

And I loved my colleagues too.  I tried my best to help them when possible (by answering questions, offloading bugs, reviewing their code, implementing helper functions they needed, writing scripts to simplify drudgery, buying unhealthy snacks for them from Costco) to make their lives a little bit easier and so that they’d have more time to work on things that they found interesting.  I never wanted them to leave (and jokingly threatened to kill some of them if they ever tried).

People in HUG helped me buy a car for the first time.  HUG filled two tables at my wedding.  HUG was a family, and our products were our babies.  I don’t know how I’m going to bear seeing them in the foster care of complete strangers.

Tony Fregoso

Tony (LinkedIn) was a member of our amazing QA team (a team that suffered its own layoffs a couple of years ago).He had both QA engineering and management roles during his time with us. QA was important to us on a personal and professional level. They kept the quality of our products high, and knew the products and their history inside-and-out.

Tony shared his memories and thoughts with us:

When I look back at my time at VMware the one word that always comes to mind is family. Beyond all of the incredible technical feats the teams achieved it is all dwarfed by the shear strength of the bonds that I formed and saw formed with the people that I worked with at VMware.

Even as the company grew and changed HUG, Desktop QA and the greater Desktop Business Unit retained much of its core identity because of the people who worked within it. The passions that were shared for the products was equally shared for the people. In the valley where it is the norm for people to change jobs ever 2 years we had a team that clearly pushed against this. Between the Dev and QA teams we had some of the most tenured members in the entire company. This happened for many reasons, a shared passion of quality, love and dedication to the products we worked on and the close bonds we had with each other.

I am thankful to VMware for bringing us all together in the way that it did, regardless of how things ended. The fact is that the strengths of the bonds that we formed are far greater and are something that will always exist.

Family

Our users had a lot to say

I was surprised by the outpouring of love from our users. I want to share a few select comments from my earlier blog post:

Bruno Kerouanton said:

I just wanted to congratulate you and all the team on the fabulous work you did. I bought my first license for VMware Workstation Linux 2.0, back in 2001 ! And use Workstation and Fusion on a daily basis (See my latest blog article on http://éé.net/ak6), it’s just a critical part of my infrastructure!

So I’m sad for you. And I just wanted to say I love you for what you made available for so many people worldwide 😉

jorgedlcruz said:

I’m a VMware vExpert because I did my Home Labs using Workstation, or even Fusion sometimes, you helped so many Companies out there, not just power users, I saw some environments using Workstation at really high scale, insane but working!

You guys did just amazing job all this time. I just can say, thank you and good luck!

Jeff:

Its really no wonder now, why apps like VMware Player and Fusion just worked so well despite doing really complicated things. Kudos to your team for really being the best champions of your product and making the computing world a much better place (this is what happens, for anyone else interested, when keeping developers happy and engaged takes precedence over keeping salesmen happy and engaged).

skimans:

Big thanks to you all 🙂 I was one of those early users. This software changed life of many people for better. Sorry to hear bad news. it’s bad move to shut down this products and your team. This software is living ad for whole company, for many of us first step into virtualisation.

R Warder:

Great product that changed the way the world works – testing and development was different before VMWare. So slow. This article was great insight into the team that made our lives better. A sad announcement but best wishes to a talented group of people.

velviavelvia:

Thanks for this tribute. I was also at VMware for 9 years, starting on the Vmkernel team that built one of the first releases of ESX, and saw it grow from a team of 200 in Stanford Research Park, with personal introductions of every new employee, and pool dunkings for folks getting married, to a big corporation of over 10k. Your team was one of the most dedicated and legendary teams at VMware. So sad to see it go.

Eddie:

I’ve been a loyal Fusion user since 2008. Fusion is what convinced several colleagues of mine to go to the Mac when they got fed up with Windows machines. I proudly buy each and every new-release license(s) because of the phenomenal quality and support that was given.

When I had problems with Fusion/Windows, the engineers actually invited me to their labs in Silicon Valley and sit next to them to work the problem out. That was support (to me) that was unheard of. I was in awe at their commitment and pride in what they did.

Gary Jones:

Absolute legends , such sad / infuriating / inexplicable / perplexing news. I’ve used VMware since day 1 and never ever looked back. Without this software I’d never have progressed anywhere near as far in my career as I did.

Kermit Vestal:

Ooooh nooo! Say it isn’t so. I was jaw droppingly amazed when I saw 1.0 and could see the VM of everything was the future of everything. Its been one of my mainstay tools ever since. Many similar free and not free tools have followed since, but none compares to the feature qualities and reliability of Workstation. Its always been ahead of its time and now we know why. So sad its been stripped of its culture and I fear its future. Thanks for telling us the rest of the story.

Thank you, everyone.

Pictures speak a thousand words

We dug around and found a bunch of pictures from our time at VMware that I thought would be fun to share.

We liked food. We had our own “Unhealthy Snack Program,” where we’d keep our group stocked with candy bars, beef jerky, sodas, etc. Sometimes you need a little sugar and caffeine when you’re battling some crazy bug. I wish I had a picture of this, but it was glorious.

We once won a waffle maker at Dave & Busters, during a group outing. Here’s Keith, making some yummy waffles for breakfast. He never made me any…

 

Picnics were always a fun way to bring the team together. Often, former members of Hosted UI would take the opportunity to show up, eat some hot dogs and catch up with the rest of us. Great way to spend a day, though we ended up talking shop more than we probably should (except for that one time we climbed trees for hours, just because we could).

 

We’ve been playing poker for years. Just casual games, nothing fancy. We’d order a pizza and play for a few hours, share some laughs. Really, we were just like the pro poker players, except a 2/7 won way more often than it should have, and things like this kept happening:

 
The table won more than I did.

 

Not all of us were gamers, but a bunch of us got together most weeks to play games of some sort. Video games, card games, board games, what have you. Smash Bros, Mario Kart, Mario Wii U, Kirby, and the Rayman games were personal favorites of mine.

This was actually on the day we IPO’d! We just got a Wii and had set up the projector for some tennis action. Man, that was a long time ago…

 

If you were getting married, or just got married, you were going in the pond. It was an old VMware tradition that we fully embraced. We amped it up a bit, though, with the introduction of costumes. You know, because your clothes were all wet, so we helpfully provided new clothes!

 

 

 

 

Birthdays are something to be celebrated! Back in the day, we’d trick people by inviting them to a meeting and surprising them with cake. Eventually people came to expect it, so then it just became cake. Oh, and an amazing birthday candle that would shoot up fire like a torch for a minute, spin around, and sing.

True story: I had my first drink at VMware, in a surprise birthday meeting. And then my second. Jocelyn insisted. I’d never been so much as buzzed before. I remember Jocelyn coming in, asking me to do something, I don’t even remember, thoroughly enjoying watching me struggle to even understand what was going on. Good times.

 

There was that one time we all got dressed up, just because. It was kind of an inverse Casual Friday. To start off, here’s some great group photos of Lee Ann on Fusion Engineering, Roshini on Fusion Performance, and Jessica on Docs.

 

In order from left to right: Surendra, Roshini, Steve, Lee Ann, David, Michael, and James. (Shame we didn’t have the whole team in this shot.)

329705_10150459281007831_715261101_o

 

Okay, terrible pun alert, straight from the Facebook post: “At VMware, our managers go APE for new releases!” (Another awesome win from Dave & Busters!)

 

Pets were always welcome in our office. This is Bodie (as a puppy — it’s been a while). We had other dogs, sometimes cats. A duck followed me into the building one day.

 

 

Can you ever truly leave the team? Might come at a price… DUN DUN DUN. (I found my entire office covered in this stuff, shortly before my last day at VMware.)

 

 

 

We once got these plasticy bookshelf things made from I think recycled milk jugs? Someone realized that they could be disassembled and reassembled, so our team, always eager to end the day on a productive note, set off to build Tetris bookshelves.

 

 

 

There was that time when we were trying to get into a company-owned pinball machine that accidentally got reset from free mode to pay mode. We weren’t about to pay $0.25! So, we spent about 3 or 4 hours trying to pick the lock with instructions from the Internet and good ol’ Hosted UI ingenuity! With the lights off. Using flashlights. Inside an office room. I swear, we’re usually smart people :/ (P.S., it did not work. We gave up and went home after 1AM. The following contraption is what we built to try to pick the lock.)

 

 

Oh, and that time we decided our IRC channel could really benefit from Microsoft Comic Chat. We had this up and running on a dedicated screen 24/7.

 

I’ll probably update this over time with more thoughts and pictures as we find them.

Thanks for walking down memory lane with me, and for all the support you’ve shown Hosted UI over the past week. 🙂

A Tribute to VMware Workstation, Fusion, and Hosted UI

Updated February 2, 2016: Once you learn about our team and who we are, come take a trip down memory lane with us!

Yesterday morning, the Hosted UI team, responsible for VMware’s Workstation and Fusion products, woke up to find themselves out of a job. These products, despite being award-winning and profitable, are probably not long for this world.

I was not directly affected, in this way at least, as I had already left VMware in 2013 to work on Review Board full-time. However, many of my closest friends were, and a product I spent 9 years of my life on may have seen its last feature.

I could talk all day about how I think we got here, losing this amazing team and these fantastic products. I could point fingers and lash out at those I blame. I could talk about how furious this all makes me.

Instead, I’m going to talk about the team and what we built — and I don’t just mean our products.

Let me tell you about our team

Hosted UI

I began working in Hosted UI on August 23rd, 2004, as a bright-eyed 20 year old freshly dropped out of college. Back then, it was a small team full of amazingly bright and passionate people, working days and nights to build a product they believed in.

The culture at that time within VMware was just so fun and energizing. People wanted to be there, and were proud of their work. Features were brainstormed over games of foosball or DDR, designs discussed over free lunches and beer bashes. In the evenings, we’d order dinner in and watch Simpsons, or whatever was on.

Company culture changed over the years, becoming more corporate and stiff, but not Hosted UI. We’d work all day, with the occasional interruption for YouTube videos or some laughs, and at night we went out and had some more. Poker nights, movie nights, video game nights. Dinners out together, sometimes several times a week.

Many people came and went over those years. The team changed, though, for a software company, a surprising number remained until the very end. Even those that left kept in touch, joining for poker nights or dinners here or there, coming to the dunkings (if you were getting married, you were going in the pond), birthday celebrations, and reunions. We formed alumni lists and kept in touch. We hung out on IRC outside of work.

Poker Night

Through deadlines and downtimes, stresses and celebrations, our team worked and played hard. We were dedicated, passionate, and if you’ll allow me, we were damn good at what we did.

I left this team two years ago, but it hasn’t really felt that way. I still saw them almost every week. Our team didn’t have to be in the same building or even the same company to stay a team.

Hosted UI may no longer exist at VMware, but that’s really VMware’s loss. They lost one of the most dedicated teams they could ever hope for, the kind of team you can’t just hire again.

We built some amazing products

Workstation

WorkstationVMware Workstation was the first VMware product (back then, it was simply known as “VMware.”). At a time when dot-coms dominated the Super Bowl and Amazon was all about books, VMware Workstation was letting pioneers in the Linux world virtualize their Windows desktop so they could run Microsoft Office instead of StarOffice.

This product evolved over the years with over 15 major releases, and more features than I can count, running on every flavor of Linux and Windows. It did this without falling prey to the bloat of most long-running products, as we focused not only on making it a more powerful product but also a more usable product.

Workstation made it easy to run complex development and testing scenarios, creating and working with several virtual environments all at once across any number of host computers. It integrated your virtual desktops with your host desktop. It let you take snapshots at different moments in the lifetime of your VM, and jump between them at will. It helped you catch defects in your software through remote debugging and CPU/memory record/replay capabilities, it helped you test complex network setups with virtual LAN devices, and it worked as a powerful front-end for VMware’s Server, ESXi, and vSphere products. And, in the end, it also helped you simply run your Windows programs on Linux, your Linux programs on Windows, or whatever you wanted.

Workstation

 

Internally at VMware, Workstation was also seen as an indispensable product, helping other teams test features and devices that would eventually become selling points on the more high-end vSphere product releases. With Workstation’s ease-of-install and ease-of-use, people could get set up in minutes and get right to work.

We loved our product. This was our baby. We took input from marketing, management, sales, customers, and so on, but in the end, we were given a lot of creative liberty over the features and design. We were also given time to address technical debt, helping to get our codebase in shape for future challenges.

Workstation with Unity

I don’t know how many awards we received, but I think it was a lot. I do know that we had so many users who loved the product we poured our souls into. That meant a lot, and kept us motivated.

It was, let’s say, a challenge getting some parts of the company to really care about the product. Workstation made a lot of money, but not the hundreds of millions the company would have preferred. This, I believe, ultimately led to yesterday’s sad outcome… Still, I’m very proud of what we built.

Fusion

Fusion

Workstation was a power user product built for Linux and Windows. In 2007, its sister product, Fusion for Mac, was released. This focused more on consumer usage, helping people run Office and other Windows apps on their Mac.

At the time, Apple had just moved to Intel processors, and were touting the ability to dual-boot between Windows and MacOS X, using a feature called Bootcamp. Fusion offered a better way by letting you run Windows and MacOS X at the same time. It was popular amongst students who needed to run Windows software for class on their shiny new MacBooks. It was popular amongst developers who needed to run or test Windows or Linux environments while on the go.

Fusion

Fusion was a very different product in some ways than Workstation, but it was also very closely related. While it didn’t focus on many of the power user features that Workstation offered, it did take many of those features and reimagine them for more casual users. It also shared much of the core code that Workstation used, meaning that features could more easily be ported across and bugs fixed just once.

Fusion was a reimagining of what Workstation could have been, built for a different time and a different audience. Like Workstation, it was also built by a group of very loyal, dedicated, brilliant people, the Fusion segment of Hosted UI.

While I never worked directly on Fusion, I did get to see features I built for Workstation make their way there, and watched as our users got to try them for the first time on the Mac. It wasn’t the product I devoted my time to, but it was one I loved, and one I still use today.

And all the others

Our small team has built quite a lot over the years. Along with Workstation and Fusion, we’ve also built:

  • Player: A slimmed-down product for simply running and interacting with VMs, without all the UI of Workstation
  • VMRC: Originally a browser plugin and an SDK for embedding virtual machines in your browser or other applications (which was transitioned to one of the teams behind ESX a couple years ago and reworked into a native VM console app launched from the browser)
  • Server: A free product built from Workstation that offered remote VM hosting and management)
  • WSX: A web-based service for running VMs natively in your browser from anywhere
  • AppCatalyst: A developer-focused, API-driven development and testing service that works with Docker

I’m pretty sure there’s more, but those are the highlights.

These, along with Workstation and Fusion, were built by a team typically no larger than about 20 people (at any given point in time).

We did good.

Time for the next adventure

VMware lost a lot of amazing people, and will be feeling that for some time to come, once they realize what they’ve done. It’s a shame. As for our team, well, I think everyone will do just fine. Some of the best companies in the Silicon Valley are full of ex-VMware members, many former Hosted UI, who would probably welcome the chance to work with their teammates again.

Workstation, Fusion, and our other products may survive in maintenance mode, or they may disappear. They may continue under a new team. It’s hard to say at this point what will happen. What I can say is that no matter what happens to them, they had an amazing run, and are something every one of us can be proud of the rest of our careers.

And we can be proud of the team, the friendships, and the strong bonds we built, now and through our next adventures.

Updated 27-January-2016 at 23:31 PM: Wow, this went viral. As of right now, we’re looking at around 40,000 unique viewers. I wrote this as a tribute for our team, and am amazed by the reaction it provoked. Everyone who loved our products and reached out to us to show your love, thank you. It means so much to us. Keep them coming!

I want to be clear that I have not worked there in years and do not have inside knowledge on what will happen to these products. I updated part of the post to make that a little more clear. VMware claims they’ll continue to exist, and I really hope that’s the case. I like to think what we built will continue to live on, and I hope VMware does it justice.

You’re not safe on the Internet (but you could be)

It’s a wonderful Friday evening. You’re out enjoying it with some friends, eating dinner at a classy Italian restaurant, telling stories and laughing together as you share a delicious bottle of wine. It’s the perfect end to the week.

As dinner wraps up, you hand the waitress the first card you grab from your wallet, a debit card, and don’t think twice. Moments later, she returns and asks you, quietly, if you have another card they can try. Looking at the debit card in her hand, you put two and two together, and a sinking feeling creeps in.

You pick up your phone and check. Sure enough, your account has a $0 balance. You just deposited your paycheck last week, and have been careful to keep a reasonable balance in there. And it’s gone. All of it.

You can’t begin to understand what happened, as you frantically call the bank, your night utterly ruined. But someone knows. The one who drained your account. The one who entered just the right username and password. The same password you use on a dozen sites. The same one you used to order flowers three years ago on a local boutique’s site. The same site whose password database was quietly hacked last week.

The Internet is a wild place

Most people on the Internet only see the tip of the iceberg. They’re on Netflix, Google, YouTube. They’re playing mobile games, ordering from Amazon. The Internet is like a giant mall with the best shops and arcades.

There’s more to the Internet than you may see. Darknets, where identities, weapons, drugs, and the darkest of pornography can be bought and sold. Cyberwarfare and espionage between countries of all sizes. Enormous “botnets,” or networks of compromised computers just like yours that are used to attack networks and crack passwords. Teams of hackers who work to find vulnerabilities in sites or in people and exploit them for amusement, financial gain, or just to make a statement.

This isn’t a reason to fear the Internet, to fear shopping online or visiting new places. This is a reason to respect it, and to be safe.

The line of fire

“But nobody will come after me!” you might think. “I’m not important. Who would target me?”

The problem with that line of thought is that it assumes a person is going after you specifically. That’s often not the case. Hackers and botnets go after many, many websites. They’re looking to hit the motherload. For instance, here’s some of the bigger sites hacked lately, and how many people were affected:

Ever use any of these? Me too.

So what happens next? Well, automated systems will harvest the results and begin trying these combinations of usernames/passwords on all sorts of different sites. Google/gmail, banks, dating sites, anywhere. And how big are these botnets? They can reach up to the 10s of millions of computers.

You are not specifically being targeted, but you’re in the line of fire. And this is going to keep happening, over and over again.

So let’s protect you from this. I’m going to teach you about three things:

  • Using passwords safely
  • Two-factor authentication, a second layer of protection and alert system
  • Identifying and ensuring secure connections to websites

And in case you’re wondering, yes, you are the target of my post. So keep reading.

Passwords: Your first line of defense

Most people on the Internet treat passwords like they’re a cute little passphrase to get into a clubhouse. We’re trained by our computers to pick something memorable to log us into our desktops, a code that “protects” you from others in the house who might want to check your e-mail. Much like a standard lock on your door protects you from your neighbors simply walking in.

Nobody ever really teaches us properly. It’s common to use your dog’s name as a password, or your birthdate, or something equally easy to remember and crack. It’s also common to use the same password or set of passwords on many different sites and services, which is how our fictitious Friday night was ruined.

This is extremely important to get right. Here are the rules for protecting yourself using passwords:

  1. Never use the same password for more than one site.
  2. Pick a long, strong password with uppercase and lowercase letters, numbers, and symbols, without any dictionary words.

That’s basically it. Seems simple enough in theory, but how do you keep track of all those passwords? You’d probably have 50 of them!

Don’t worry, there are tools that make this super easy.

1Password

This is one of my favorite tools for staying secure.

1Password is a tool for keeping track of your accounts and generating new passwords. It works with your browser and remembers any password you enter, and makes it really easy to fill in passwords you’ve already entered.

When you’re creating a new account or changing passwords, it’ll help you by generating strong, secure, nearly uncrackable passwords. For example, here’s one it just made for me: wXXgVzb8Zp(zwmjG7zBGkg=iT.

It’s available for Mac, Windows, iPhone, iPad, and Android. It’s free for iPhone, iPad, and Android, and costs only $35 for Mac and Windows (which is a bargain for what you’re getting).

Let me show you how it works.

I’m about to log into an account on Facebook.

Instead of typing my login and password, I just click that little keyhole icon next to the address bar, which will pop up any 1Password entries I have for Facebook.

Once I click “Facebook,” it’ll fill in my username and password and log me in. It’s just that simple.

When you’re signing up for a new account, use the Password Generator, like so:

Once you create the account, 1Password will remember the password for later. Look at that thing. Nobody’s cracking that!

This all works for any site, and even works on your iPad/iPhone:

1Password is great for more than passwords. It can keep secure notes, information on your credit cards or bank accounts, or really anything else you have that you want available but locked away.

In order to access your stuff in 1Password, you only need to remember a single password of your choosing. Make sure this is a strong one, and that you don’t forget it! Write it down if you have to.

Buy 1Password and use it. Look, $35 is nothing compared to the potential fallout of being involved in the next several major security breaches.

LastPass

LastPass is an alternative to 1Password, and is great if you’re an Android user. Usage is pretty similar to 1Password.

I don’t personally have a lot of experience with LastPass, but a lot of people love it. You can learn more about it on their site.

Pen-and-paper password journal

I’m sure you’ve been told before that it’s a bad idea to write down your passwords, am I right? Afterall, if you write them down, then gee, anyone can get them!

That’s not entirely true. The fact is, you’re more at risk reusing one or two weak passwords on the Internet than keeping dozens of strong passwords written down on paper in your home.

Let’s be clear, this is not the best option, and if you’re going to do this, keep it secret, keep it safe. Still, it’s better than not having strong passwords. If at all possible, use 1Password or LastPass, but if you absolutely must, write them own on a dedicated journal that you can keep safe somewhere. Don’t lose it, and always use it for every password.

Two-Factor Authentication: Second line of defense

You’re now using stronger passwords, and you have friendly tools to help manage them. Good job! The next step is making sure that only you can log into your most important websites.

Many sites and services (Google, Apple, Dropbox, Evernote, Bank of America, Chase, and many others) offer an extra security layer called “two-factor authentication.” This is a fancy term for “We’ll only log you in if you enter a code we’ll send to your phone.” When enabled, these services will require something you know (your login/password), and something you have (your phone).

Let’s take Google, for example. You can set things up so that after entering your username and your brand-new secure password, Google will send you a text message with a 6-digit code. Once you receive the text (which only takes a second), you’ll enter it on the site, and you’ll be logged in.

Now why would you do that? To prove that you are the one logging in, and not someone who’s figured out your password. The cell network’s going to make sure that text is only going to your phone, and you’re going to prove it’s you logging in. (Some services will require that you use a specialized app, or a hardware device that fits on your keychain, but most will work with text messages.)

Imagine someone did get a hold of your password, and tried logging in. You’re still going to get that text, but that person will not. He/she won’t be able to log in as you. You’re safe! It’s also going to be a dead giveaway that your password was compromised. You’ll want to change it right away.

Basically, this is both your gated community and your alarm system.

This is pretty easy to set up at most places. Here are some guides:

You can find more at twofactorauth.org. Just click the icon under “Docs” for any service you use that’s green.

I’ll be honest with you, this will feel unfamiliar at first, and you might be tempted not to do it. Trust me, though, this is worth turning on for as many services as possible. You’ll be glad you did next time one of these companies announces a security breach.

Identifying and ensuring secure websites

Let me briefly explain how the Internet works.

When you connect to a website, the browser will usually try to access it first over the “HTTP protocol.” This is the language that browsers and web servers speak. This communication is in plain text, which means anybody that listens in can read what’s being posted. This is sort of like handing a piece of paper to someone, and having them pass it along to the next person, and so on, until it reaches its final destination.

That’s very bad if you’re sending anything confidential. Passwords, for instance. It’s important that you learn to identify when you’re on a secure website.

You know the address/search bar on your browser? Look to the left. If you see http://, or you don’t see anything but an icon and a domain name, you’re on an insecure website.

However, if you see https://, or a lock icon, or a green banner, you’re good! This is using HTTPS, an encrypted connection, meaning that nobody in-between can listen in. That’s more like writing your letter in gibberish that only you and the final recipient understand.

For comparison, these are secure:

This is not:

Always look for these before filling out any forms. If it’s not showing a green banner or lock, you don’t want to give the site any sensitive information.

If you see a green banner, it’ll show the name of the company or organization. This is showing that the encrypted channel and website have been verified by a “certificate authority,” an entity that issues certificates for these encrypted channels. It means they’ve checked that, for instance, ally.com is owned by Ally, and not by someone pretending to be Ally.

If you click the banner or the lock icon, you’re going to see some more information about the connection. Most of this will be highly technical, but you should see some blurbs about who verified the authenticity of the site, and some information on the organization owning the site.

Most websites these days are moving to encrypted HTTPS connections, and most will automatically redirect all requests from HTTP to HTTPS. This is good, but you can go a step further and have your browser always start out using the HTTPS connection whenever possible. This takes almost no effort, and is worth doing.

This is done with a browser extension called HTTPS Everywhere.

If you’re running Chrome as your browser, simply install it from the Chrome store.

If you’re running Firefox, install it from the Firefox add-on store.

Did you do it? Great, you’re done! You’re now a little bit safer on the Internet!

Putting it all together

I threw a lot of information at you, but hopefully you’ve learned a lot and will put it into practice.

So let’s summarize. If you follow the above, you’ll:

  • Be at less risk for identity and financial theft the next time there’s a major security breach, since your passwords won’t be shared.
  • Be alerted when someone tries to log in as you on any service with two-factor authentication enabled.
  • Have passwords strong enough to be unguessable and nearly uncrackable, for all sites and services you use.
  • Know how to identify secure websites, so you don’t leak passwords or other private information to anyone who’s listening in.
  • Automatically connect to the most secure version of a website whenever possible.

Not bad for a little bit of work. Hopefully by now, you realize that this does matter, because you, I, and everyone else really is a target, simply because we’re all part of something large enough to be a target.

So pass this around. Tell your friends about what you’ve learned. Educate your kids. Stay safe on the Internet.

A better web through spreadsheets

I’ve spent the past couple of days basically living in spreadsheets, crunching sales data, entering equations, building pivot tables and forecasts, and painstakingly toggling cell borders… Your typical spreadsheety stuff. (And I didn’t go crazy at all.)

While spreadsheet work is a task an engineer would often dismiss, loathe, and try to pawn off onto an intern or manager, I’ve come to realize the opportunity we as an industry have missed.

 

A world on a web

The World Wide Web has been a part of most people’s lives for a couple of decades now. It has transformed society, and we take it for granted today. Before the web, communication wasn’t quite so pleasant. We had to visit our friends in person if we wanted to talk or play a game together. The events of a wild party stayed mostly in the minds of the participants, and couldn’t easily be shared with millions of people around the world. We didn’t even know that cats and cheeseburgers went so well together.

That's not what I meant!
That’s not what I meant!

It was the dark ages, and frankly, we should be embarrassed to even talk about it.

Then a wonderful man named Sir Tim Berners-Lee created the Web. There were probably other people involved, but it doesn’t matter really. The point is, he did a pretty great job and we should all buy him a drink if he’s in town.

Let me briefly explain how the web works on a technical level, using a common analogy of computers as tactical submarines. Imagine you’re in a US submarine (your computer) and you want to get some cat pictures from the guy in the Russian submarine (a Russian server). You know where in the sub he is (a “URL”), and know that the only way to get to him is through an unsecured port (we call this a “HTTP port”) or a mostly-secured-but-sometimes-not port (“HTTPS port”).

You’d load a torpedo with a letter asking for cat pictures (these are “packets”) and fire it off through their port (“HTTP/HTTPS”) into the location of the guy with the pictures (“URL”). Being trained to handle this, the torpedo would be intercepted, a new one stuffed with cat pictures, and fired back at your submarine.

This is the primary use of the web. Not so much torpedoes.
This is the primary use of the web. Not so much torpedoes.

That’s… basically how the web works.

Oh and there’s also HTML. This is the universal language of web pages. It comes with a family of other technologies, like CSS, JavaScript, VBScript, Dart, Silverlight, Flash, Adobe Flex, Java, ActiveX, and a myriad of innovative plugins.

Where was I? .. Oh yeah, spreadsheets.

(Spreadsheets are more like Battleship. A5! B12!)

 

The missed opportunity

We have built the world’s communication, social interaction, and repositories of cat pictures on top of the web, and therefore HTML (and co).

What I’ve realized over the past two days is that building it on top of HTML was a mistake. We should have built it on top of spreadsheets.

We could have had this!
We could have had this!

Hear me out.

Spreadsheets have been around a long time, and unlike HTML/CSS/JavaScript, people just naturally understand them. They’re simple, intuitive, and fun!

In the dark times before tab-based browsing, a time when browser manufacturers thought window management should most resemble the winning animation of Solitaire, Spreadsheets had multiple tabs. The right technology coud have put us years ahead of where we are now.

As developers, we face religious wars over table-based layouts vs. non-table-based layouts. We waste thousands of man years on this. Spreadsheets, being nothing but table-based, would have saved us all a whole lot of trouble.

It took a long time for the world to realize JavaScript could be used for more than scrolling status bar updates and trailing mouse cursors; it could be used to write useful things, like Facebook and Twitter! All the while, spreadsheet power users were writing complicated macros to do anything they could ever want. I mean, look at this guy who wrote a freaking RPG in Excel!

Look at those graphics.
Look at those graphics. Look at them.

Spreadsheets are inherently social. You can save them, edit them, pass them out to your friends. You can’t do that with your Facebook wall. Ever try to save or edit someone else’s webpage? Yeah, I bet that worked out great for you.

Developers, how many different third-party APIs are you dealing with in order to generate some meaningful statistics and reports for your app/startup? How much money are you paying to generate those reports? How much code did you have to write to tie any of this together? In the spreadsheet world, you’d just stick some pivot tables and graphs on the page and call it a day, spend some time with your family.

None of this nonsense with disagreements between slow-moving standards bodies that keep going back-and-forth on everything. Instead, I think we’d all feel comforted knowing we could leave this all in the hands of Microsoft.

 

What can you do…

I know, I know. It seems so obvious in retrospect. I guess all I can say on their behalf is that the web was once a new, experimental project, and such things are rarely perfect. Even my projects have some flaws.

Sir Tim, call me. We’ll get this right next time.

Remembering the Line Ride

I spent the holidays at Disneyland this year with my girlfriend and my family. We stood in numerous lines for hours on end during the busiest week of the year, waiting to see Disney’s take on classic rides such as the Haunted Mansion and Small World.

Their take was fantastic, but this post is not about that.

Standing in line for the Haunted Mansion, listening to people murmur about how agonizing the lines were, it dawned on me that not everybody understood nor appreciated the true origins of these amazing amusement parks. My sister certainly didn’t know, and neither did my girlfriend.

You may not either, so allow me to share a bit of history.

Back to the middle ages

Much of what we’ve come to enjoy in amusement parks originated from fairs in the Middle Ages [1]. The food, the shows. They were further inspired over time by other events and inventions throughout the centuries that followed. One of the innovations in amusement technology that really sparked the modern era of amusement park rides was a classic mechanical ride, the steam-powered carousel, built by Thomas Bradshaw at the Alysham Fair in 1861 [2].

The problem with technological innovations is that they overshadow the simpler pleasures that came before them.

The Line Ride

Long before the carousel, in 1733, people enjoyed a simpler tradition. The humble fairgrounds in those days were unlike the marvels we have today, but were still full of events for children and adults of all ages.

One of the most beloved traditions in those days was known as the Rope Line Ride, or the Line Ride for short. Long lines of rope, attached to tall stakes in the ground, would be laid out in all sorts of patterns, forming paths for the kids to traverse. Common patterns included the spiral, the back-and-forth, and the weave.

Participating in the Line Ride was simple. A person would start at one end, following the line, seeing where it took them (by a garden, perhaps, or a wall of funny drawings), eventually coming out on the other side.

Remember, these were the days when Kick the Can and Hoop Rolling were the rage. The Line Ride was so popular that it was often nearly full of people, but this gave them time to socialize and join together in the admiration of their surroundings.

Evolution of the Line Ride

Times change, as they often do. While once a fun and common attraction, the younger generations began to grow weary of the Line Ride. In 1861, Thomas Bradshaw, the aforementioned inventor of the steam-powered carousel, forever changed the Line Ride by making it a means to an end. He put the carousel at the very end of the Alysham Fair Line Ride.

Now, instead of simply enjoying the Line Ride for what it was, people were passing through it, with great impatience, just to get to the all-new steam-powered carousel.

A new tradition was born. The Line Ride no longer became an attraction itself, but rather simply the Line, a way to control the flow of people leading up to an attraction. This was seen as a very controversial change in its day — after-all, the Line Ride was a tradition going back over a hundred years — and with it came a distrust of the newer attractions by the older generations. Of course, time passed, and the Line became the norm.

The spirit carries on

While often forgotten as an attraction, the Line Ride’s spirit remains today in our terminology and our parks. We’re all familiar with celebrities walking down the rope line, or hearing about people “working the rope line.”

And, of course, the long, grueling lines leading up to the popular attractions at amusement parks and carnivals around the world.

Using Freshdesk with PagerDuty for Better Customer Support

At Beanbag, we’ve been using Freshdesk to handle customer support for Review Board, Power Pack, and RBCommons.

We’ve also been using PagerDuty to inform us on any critical events, such as servers going down, memory/CPU load, or security updates we need to apply to our servers.

Our customers’ problems are just as important to us as our servers’ problems, but we were lacking a great way to really get our attention when our customers needed it most. After we started using PagerDuty, the solution became obvious: Integrate PagerDuty with Freshdesk! But how? Neither side had any native integration with the other.

Enter Freshdesk webhooks

Freshdesk’s webhooks support is pretty awesome. Not only can you set them up for any custom condition you like, but payload they send is completely customizable, allowing you to easily construct an API request to another service – like PagerDuty!

This is super useful. It means you won’t need any sort of proxy service or custom script to be set up on your server. All you need are your Freshdesk and PagerDuty accounts.

Deciding on your setup

There are probably many ways you can configure these two to talk, and we played with a couple configurations. Here are the general rules we settled on:

  • Only integrate PagerDuty for paying customers (with whom we have support contracts). We don’t want alerts from random people e-mailing us.
  • When a paying customer open new tickets or reply to existing tickets, assign them to a “Premium Support” group, and create an alert in PagerDuty.
  • When an agent replies or marks a ticket in “Premium Support” as resolved, resolve the alert in PagerDuty.

We always resolve the alert instead of acknowledging it, in order to prevent PagerDuty from auto-unacknowledging a period of time after the agent replies. When the customer replies, it will just reuse the same alert ID, instead of creating new alerts.

Also note that we’re setting things up to alert all of our support staff (all two of us founders) on any important tickets. You may wish to adjust these rules to do something a bit more fine-grained.

Okay, let’s get this set up.

Setting up PagerDuty

We’re going to create a custom Service in PagerDuty. First, log into PagerDuty and click “Escalation Policies” at the top. Then click the New Escalation Policy button.

Name this policy something like “Premium Support Tickets,” and assign your agents.

Next, click “Services” at the top. Then click the Add New Service button and set these fields:

title

Click Add Service. Make a note of the Service API Key. You’ll need this for later.

Edit your service and Uncheck the Incident Ack Timeout and Incident Auto-Resolution checkboxes. Click Save.

Optionally, configure some webhooks to point to other services you want to notify. For instance, we added Slack, so that we’ll instantly see any support requests in-chat.

Okay, you’re done here. Let’s move on to Freshdesk.

Setting up Freshdesk

Freshdesk is going to require four rules: One Dispatch’r and three Observers.

I’ll provide screenshots on this as a reference, along with some code and URLs you can copy/paste.

Start by logging in and going to the Administration page.

Dispatch’r Rule: Alert PagerDuty for important customer tickets

Click “Dispatch’r” and add a new rule.

Dispatchr rule

Set the Rule name and Description to whatever you like. We added a little reminder in the description saying that this must be updated as we add customers.

For the conditions, we’re matching based on company names we’ve created in Freshdesk for our customers. You may instead want to base this on Product, From E-mail, Contact Name, or whatever you like.

For the Callback URL, use https://events.pagerduty.com/generic/2010-04-15/create_event.json. Keep note of this, because you’ll use it for all the payloads you’ll set.

Now set the Content to be the following.

{
    "service_key": "YOUR SECRET KEY GOES HERE",
    "event_type": "trigger",
    "description": "Ticket ID {{ticket.id}} from {{ticket.requester.company_name}}: {{ticket.subject}}",
    "incident_key": "freshdesk_ticket_{{ticket.id}}",
    "client": "Freshdesk",
    "client_url": "{{ticket.url}}",
    "details": {
        "ticket ID": "{{ticket.id}}",
        "status": "{{ticket.status}}",
        "priority": "{{ticket.priority}}",
        "type": "{{ticket.ticket_type}}",
        "due by": "{{ticket.due_by_time}}",
        "requester": "{{ticket.requester.name}}",
        "requester e-mail": "{{ticket.from_email}}"
    }
}

Make note of the whole YOUR SERVICE KEY GOES HERE part in line 2. Remember the service key in PagerDuty? You’ll set that here. You’ll also need to do this for all the webhook payloads I show you from here on out.

Go ahead and add any other actions you may want (such as adding watchers, or setting the priority), and click Save.

Click Reorder and place that rule at the top.

Setting up Freshdesk Observer

Now we need to set up a few observers. Go back to the Administration page and click “Observer.” We’ll be adding three new rules.

Observer Rule #1: Resolve PagerDuty alerts on close

Add an event. You’ll set:

Resolve on Close rule

Use the same Callback URL as earlier, and set the Content to:

{
    "service_key": "YOUR SERVICE KEY GOES HERE",
    "event_type": "resolve",
    "description": "{{ticket.agent.name}} resolved ticket {{ticket.id}}: {{ticket.subject}}",
    "incident_key": "freshdesk_ticket_{{ticket.id}}",
    "details": {
        "ticket ID": "{{ticket.id}}",
        "status": "{{ticket.status}}",
        "priority": "{{ticket.priority}}",
        "type": "{{ticket.ticket_type}}",
        "due by": "{{ticket.due_by_time}}",
        "requester": "{{ticket.requester.name}}",
        "requester e-mail": "{{ticket.from_email}}"
    }
}

Don’t forget that service key!

Observer Rule #2: Resolve PagerDuty alerts on agent reply

Let’s add a new event. This one will resovle your PagerDuty alert when an agent replies to it.

Resolve on Reply rule

Again, same Callback URL, with this Content (and your service key):

{
    "service_key": "YOUR SERVICE KEY GOES HERE",
    "event_type": "resolve",
    "description": "{{ticket.agent.name}} acknowledged ticket {{ticket.id}}: {{ticket.subject}}",
    "incident_key": "freshdesk_ticket_{{ticket.id}}",
    "details": {
        "ticket ID": "{{ticket.id}}",
        "status": "{{ticket.status}}",
        "priority": "{{ticket.priority}}",
        "type": "{{ticket.ticket_type}}",
        "due by": "{{ticket.due_by_time}}",
        "requester": "{{ticket.requester.name}}",
        "requester e-mail": "{{ticket.from_email}}"
    }
}

Observer Rule #3: Alert PagerDuty on customer reply

title

Here’s your Content:

{
    "service_key": "YOUR SERVICE KEY GOES HERE",
    "event_type": "trigger",
    "description": "{{ticket.agent.name}} acknowledged ticket {{ticket.id}}: {{ticket.subject}}",
    "incident_key": "freshdesk_ticket_{{ticket.id}}",
    "details": {
        "ticket ID": "{{ticket.id}}",
        "status": "{{ticket.status}}",
        "priority": "{{ticket.priority}}",
        "type": "{{ticket.ticket_type}}",
        "due by": "{{ticket.due_by_time}}",
        "requester": "{{ticket.requester.name}}",
        "requester e-mail": "{{ticket.from_email}}"
    }
}

Done!

You should now be set. Any incoming tickets that match the conditions you set in the Dispatch’r rule will be tracked by PagerDuty.

Now you have no excuse for missing those important support tickets! And your customers will thank you for it.

Breaking back into your network with the Synology Web UI

Have you ever left town, or even just took a trip to the coffee shop, only to find that you’re locked out of your home network? Maybe you needed a file that you forgot to put in Dropbox, or felt paranoid and wanted to check on your security cameras, or you just wanted to stream music. I have…

The end of a long drive

Last night, I arrived at my hotel after a 4 hour drive only to find my VPN wasn’t working. I always VPN in to home, so that I can access my file server, my VMs, security cameras, what have you. I didn’t understand.. I was sure I had things set up right. You see, I recently had my Xfinity router replaced, and had to set it up to talk to my Asus N66U, but I was absolutely sure it was working. Almost sure. Well, I thought it was working…

So I tried SSHing in. No dice. Hmm.. Any web server ports I exposed? Guess not. Maybe port forwarding was messed up somewhere?

Ah HA! I could reach my wonderful Synology NAS’s web UI. If you haven’t used this thing, it’s like a full-on desktop environment with apps. It’s amazing. Only thing it’s really missing is a web browser for accessing the home network (get on this, guys!). After spending some time thinking about it, I devised a solution to get me back into my home network, with full VPN access (though, see the end of the story for what happened there).

Christian’s step-by-step guide to breaking in with Synology

No more stories for now.

To get started, I’m assuming you have three things:

  1. Remote access (with admin rights) to your Synology NAS’s web console.
  2. A Linux server somewhere both sides can log into remotely (other than your local machine, as I’m assuming yours isn’t publicly connected to the network).
  3. A local Linux or Mac with a web browser and ssh. You can make this work on Windows with Putty as well, but I’m not going into details on that. Just figure out SSH tunneling and replace step 7 below.

All set? Here’s what you do.

  1. Log into your NAS and go to Package Center. Click Settings -> Package Sources and add:
  2. Name: MissileHugger
    Location: http://packages.missilehugger.com/
  3. Install the “Web Console” package and run it from the start menu.
  4. Web Console doesn’t support interactive sessions with commands, so you’ll need to have some SSH key set up on your linux server’s authorized_keys, and have that key available to you. There’s also no multi-line paste, so you’ll need to copy this key through Web Console line-by-line:

    Locally:

    $ cat ~/.ssh/id_dsa

    On Web Console:

    $ echo "-----BEGIN DSA PRIVATE KEY-----" > id_dsa
    $ echo "<first line of private key>" >> id_dsa
    $ echo "<second line of private key>" >> id_dsa
    $ ...
    $ echo "-----END DSA PRIVATE KEY-----" >> id_dsa
    $ chmod 600 id_dsa
  5. Establish a reverse tunnel to your Linux box, pointing to the web server you’re trying to reach (we’ll say 192.168.1.1 for your router).

    Remember that Web Console doesn’t support interactive sessions, or pseudo-terminal allocation, so we’ll need to tweak some stuff when calling ssh:

    $ ssh -o 'StrictHostKeyChecking no' -t -t -i id_dsa \
          -R 19980:192.168.1.1:80 youruser@yourlinuxserver

    The ‘StrictHostKeyChecking no’ is to get around not having any way to verify a host key from Web Console, and the two -t parameters (yes, two) forces TTY allocation regardless of the shell.

  6. If all went well, your Linux server should locally have a port 19980 that reaches your web server. Verify this by logging in and typing:
    $ lynx http://localhost:19980
  7. On your local machine, set up a tunnel to connect port 19980 on your machine to port 19980 on your Linux server.
    $ ssh -L 19980:yourlinuxserver:19980 youruser@yourlinuxserver
  8. You should now be able to reach your router. Try it! Open your favorite browser and go to http://localhost:19980
  9. Clean up. Delete your id_dsa you painfully hand-copied over, if you no longer need it, and kill your SSH sessions.

Epilogue

While this worked great, and I was able to get back in and see my router configuration, I wasn’t able to spot any problems.

That’s when I realized my Mac’s VPN configuration was hard-coding my old IP address and not the domain for my home network. Oops 🙁

Hope this helps someone!

A new adventure begins

Act 1, Scene 1

August 23rd, 2004. A young kid, not even 21, freshly dropped out of college, passionate about open source and programming. He walks into his new office at his new job at VMware, his first job, ready to start the day, eager to impress and meet his new co-workers.

Nobody was there. Thumbs twiddled.

10AM starts to roll around, and finally, the first sign of life. Over the next couple hours, more people show up.

Over the next week, he’s set up and learning the ropes. Working on his first bug, soon his first feature. Attending his first team get-togethers. Making his first Bay Area friends.

Over the next few months, his first birthday celebration at work. His first glass of champagne. His first real responsibilities.

Over the next few years, bigger roles, leadership roles. He began to get a feel for where he’s truly going in this silly little world.

This, of course, was me, on my first adventure in the tech industry.

I was lucky to be placed in a fantastic team full of smart, hard-working, dedicated, and fun software engineers and managers. We’d discuss architecture, brainstorm ideas, joke around, watch YouTube videos, play poker, watch movies, go to events. The web of awesome people extended throughout the company as well.

Over the past nine years, I worked on a great many things.

  • Eight releases of VMware Workstation, including a three-year effort to build Workstation 8.0 (a major undertaking).
  • VMware Server 1.0. I was the primary Linux developer, pulling caffeine-fueled all nighters to meet insane deadlines.
  • Player and VMRC, which powers the VM console for our enterprise products.
  • The core foundation used in Fusion and other products.
  • Icons and artwork for the Linux products.
  • I introduced Unity to Workstation. (Sorry, guys…)
  • Helped in the creation of the current generation of the View client for Linux.
  • More recently, I developed WSX, an experiment in developing a pure web client and console for accessing remote VMs anywhere, from desktops and tablets.

Not a bad run.

This Thursday, August 1st, 2013, I’ll be leaving VMware.

Revision 1: “Add the reviewboard”

Several years ago, I began working with my good friend David Trowbridge on an open source project for keeping track of patches and easing the review process. We spent many years in the open source world looking at raw diffs on bug trackers and in e-mails, and things weren’t that much better at VMware. As Mr. Wonderful says, “There has to be a better way!”

So we slaved away in the late nights and weekends, iterating and iterating until we had something we could use. We named this product “Review Board” (or “the reviewboard,” as our first commit says). We put it out there for people to play with, if anyone was interested.

There was interest. Review Board is now used around the world at companies big and small. We’ve continued to improve and grow the product and turn it into something that developers actually want to use.

We later built a startup around this. Beanbag.

It’s dangerous to go alone. Take this.

Earlier this year, we met a local entrepreneur as part of a program we participate in. We quickly developed a rapport, and he offered to help and advise us in our efforts to grow our business. It wasn’t long after that we started discussing funding, and where that could get us.

We started pitching, and he reached out to his contacts. Before long, we had what we needed to give this a try for a couple years.

Step 3: Profit?

There’s a lot of hard work ahead of us, but we’re up to the challenge. It’s both exciting and terrifying.

Leaving my team behind at VMware is hard, but everyone has been so supportive.

IMG_0720

Basically.

In the coming months, Review Board’s going to grow in exciting new ways. We’ll be gearing up for a new 1.8 release, releasing our first commercial extension to Review Board, and improving our SaaS, RBCommons. We have a pretty good idea where we want to go from here, and now we can better focus on making it happen.

It’s going to be an awesome adventure.

WSX 1.1 beta is released, with bookmarkable VMs!

WSX

I’ve been pretty quiet on the WSX front since the release of WSX 1.0. A lot of work has been put into taking this from a prototype to something more solid, more functional.

Yesterday, we released a beta of WSX 1.1, which takes a big step in that direction, improving the reliability and access to your VMs, with a couple new features. Let’s go through them!

 

Bookmarkable VMs

To get to your favorite VM before, you’d have to connect to WSX and navigate to it every time, which was.. kind of a pain. No more, I say! Each VM now has its own URL, and that URL is bookmarkable. Place a bookmark in your browser’s toolbar for quick access, or bookmark to the home screen on your iPad.

Sure, you’ll have to log in if it’s been a while, but you won’t have to navigate all the way to your VM every time. As for the annoyance of constantly logging in to your servers…

 

Persistent Server Connections

Every new tab or reload disconnected all your server sessions before, due to how we mapped a browser’s connection to a server’s connection. That’s been made a lot smarter in 1.1. Now, once you connect, you can open as many other tabs/windows to WSX as you want and they’ll share your server sessions. You can even close all your tabs, and so long as you open WSX again within 5 minutes, you won’t have to log in again.

That means you can log in to a server and open each VM you want to work with in their own tabs without logging in more than once. Cool, right? Really handy for bookmarkable VMs.

 

Other Enhancements

Those were the two big features, but there were lots of other enhancements and fixes. In general:

  • New icon!
  • Various cursor and key fixes for Internet Explorer
  • Faster graphics performance
  • Key repeat now works
  • Caps Lock improvements

 

Get it while it’s hot!

Weird bugs: Django, timezones, and importing from eggs

Every so often you hit a bug that makes you question your sanity. The past several days have been spent chasing one of the more confusing ones I’ve seen in a long time.

Review Board 1.7 added the ability to set the server-wide timezone. During development, we found problems using SSH with a non-default timezone. This only happened when updating os.environ[‘TZ’] to something other than our default of UTC. We’d see the SSH process (rbssh, our wrapper for SSH communication) break due to an EOF on stdin and stdout, and then we’d see the development server reload itself.

Odd.

Since this originated with a Subversion repository, I first suspected libsvn. I spent some time going through their code to see if a timezone update would break something. Perhaps timeout logic. That didn’t turn up anything interesting, but I couldn’t rule it out.

Other candidates for suspicion were rbssh itself, paramiko (the SSH library), Django, and the trickster god Loki. We just had too many moving pieces to know for sure.

So I wrote a little script to get in-between a calling process and another process and log all communication between them. I tested this with rbssh and with plain ol’ ssh. rbssh was the only one that broke. Strange, since it wasn’t doing anything obviously wrong, and it worked with the default timezone. Unless it was Paramiko somehow…

For the heck of it, I tried copying some of rbssh’s imports into this new script. Ah-ha! It dropped its streams when importing Paramiko, same as rbssh. Interesting. Time to dig into that code.

The base paramiko module imports a couple dozen other modules, so I started by narrowing it down and reducing imports until I found the common one that breaks things. Well that turned out to be a module that imported Crypto.Random. Replacing the paramiko import in my wrapper with Crypto.Random verified that that was the culprit.

Getting closer…

I rinsed and repeated with Crypto.Random, digging through the code and seeing what could have broken. Hmm, that code’s pretty straight-forward, but there are some native libraries in there. Well, all this is in a .egg file (not an extracted .egg directory), making it hard to look through, so I extracted it and replaced it with a .egg directory.

Woah! The problem went away!

I glance at the clock. 3AM. I’m not sure I can trust what I’m seeing anymore. Crypto.Random breaks rbssh, but only when installed as a .egg file and not a .egg directory. That made no sense, but I figured I’d deal with it in the morning.

My dreams that night were filled with people wearing “stdin” and “stdout” labels on their foreheads, not at all getting along.

Today, I considered just ripping out timezone support. I didn’t know what else to do. Though, since I’m apparently a bit of a masochist, I decided to look into this just a little bit more. And finally struck gold.

With my Django development server running, I opened up a separate, plain Python shell. In it, I typed “import Crypto.Random”. And suddenly saw my development server reload.

How could that happen, I wondered. I tried it again. Same result. And then… lightbulb!

Django reloads the dev server when modules change. Crypto is a self-contained .egg file with native files that must be extracted and added to the module path. Causing Django to reload. Causing it to drop the spawned rbssh process. Causing the streams to disconnect. Ah-ha. This had to be it.

One last piece of the puzzle. The timezone change.

I quickly located their autoreload code and pulled it up. Yep, it’s comparing modified timestamps. We have two processes with two different ideas of what the current timezone is (one UTC, one US/Pacific, in my case), meaning when rbssh launched and imported Crypto, we’d get a bunch of files extracted with US/Pacific-based timestamps and not UTC, triggering the autoreload.

Now that the world makes sense again, I can finally fix the problem!

All told, that was about 4 or 5 days of debugging. Certainly not the longest debugging session I’ve had, but easily one of the more confusing ones in a while. Yet in the end, it’s almost obvious.

Scroll to Top