THE OMNI SHOW

Brent Simmons: You're listening to The Omni Show. Get to know the people and stories behind The Omni Group's award winning productivity apps for Mac and iOS. Muuusic.

[MUSIC PLAYS]

Brent: I'm your host, Brent Simmons, and this is a special episode. We're talking about how we built OmniFocus for the Web. In fact, we're probably still building it, but we'll go into how we did it.

We have a big crowd today. In the studio with me today is Ken Case, CEO of The Omni Group. Grayson West, design manager. Chris Pruitt, web developer. And Mike Davies, DevOps engineer / guru. Say hello, Ken.

Ken Case: Hello Ken.

Brent: Hello Grayson.

Grayson West: Hello Grayson.

Brent: Hello Chris.

Chris Pruitt: Hello Chris.

Brent: Hello Mike.

Mike Davies: Hello Mike.

Brent: Thanks guys. Alright. So let's go way back. It turns out that this project started nine years ago. Ken, tell us about the early days of OmniFocus for the Web. I assume then it was all Flash and ActionScript.

Ken: We started OmniFocus for the Web about the same time as we started working on OmniFocus 2, actually. So shortly after we shipped OmniFocus 1 we started hearing a lot from customers that were using OmniFocus and trying to use it at work, but they weren't necessarily able to bring their Mac to work. And so they were looking for something else.

So we started building OmniFocus for the Web in 2009, hired a full time web developer to come on board the project with us. We started evaluting the various technologies that were available to us at that time like Cappuccino and SproutCore.

Brent: I remember those well. Cappuccino was the one that looked like Objective-C, right?

Ken: Yeah it was written in a language called Objective-J, and it tried to make everything work exactly like it would work in a Cocoa app. So it had AppKit, it had scroll views. It had all these sorts of things that a native app would have.

Ken: And I believe they're approaching 1.0 right now, actually — this month, so.

Brent: Amazing, I thought I heard those guys went to Apple. Maybe someone else picked it up. I don't know.

Ken: Maybe they didn't. Yeah. It's just the open source project that's continued on.

Brent: Who was the engineer that we hired? Was that Andrew Burkhalter?

Ken: Yes, that's right.

Brent: Okay, and he's still here, working on OmniFocus.

Ken: Yeah, absolutely.

Ken: So if we started this nine years ago, why has it taken so long to get there? So I mentioned that we started this as we were also working on OmniFocus 2. Of course, 2008 was the year the App Store became available on the iPhone. So this was the shortly afterward.

But the big interruption came the next year, in 2010 when, in January, Apple announced the iPad. And we decided we were going to bring all of our apps to the iPad. So we put a lot of projects on hold for that. Both some of the big Mac updates that we were working on and of course, this project.

Brent: I remember famously Omni said iPad or Bust! Exclamation mark. At that time. It's an old blog post, even.

Brent: So iPad got all... well, a lot of the attention at that time. Had we gotten as far as prototypes? Was the thing working at all?

Ken: Oh yeah, yeah. We had prototypes working with drag and drop and all sorts of stuff going on. But we had to... after evaluating several different options we ended up going with Cappuccino. But we were pushing the edges of what it supported at that time. Then we ended up sort of extending it and trying to… fixing bugs with it. It started slowing down after the initial fast progress.

Brent: It's always fun to get that first prototype. If you're like me you think to yourself, “Wow. Geez. I'm almost done! Look!” And then that's when things get tough.

Ken: Yeah we were very excited by some of the navigation things that we had come up with.

Grayson: Yeah it was pretty cool. We were all pretty excited.

Ken: That was not where we ended up going.

Grayson: Then we got more excited about something else.

Brent: So that was 2009 era. What got us to reboot this project? Grayson, what did you do to make this happen again?

Grayson: So this product is the most requested bug in our bug data base. I don't know—

Brent: Across all apps?

Grayson: Across everything. And so when I became manager I periodically would bug Ken about this in our one-on-ones. You know, “when are we gonna do OmniFocus for the Web?” It's one of my saved bug searches. I would get notifications, probably… I want to say every day, but it was probably every other day. And I know Brian and the other managers were bugging Ken about this periodically.

Brent: Ken were you also bugging yourself about it periodically?

Ken: No I had plenty of people bugging me.

Brent: Alright.

Grayson: We were kind about it.

Brent: Yeah. Yeah.

Grayson: And so, about over a year... has it been over a year? It's been over a year, right Chris?

Chris: Uh yeah. Just about thirteen, fourteen months....

Grayson: Finally in a one-on-one with Ken, I marked it off, my OmniFocus action of bug Ken about OmniFocus for the Web. And Ken's like, “Okay, let's do it.” I think that's what your words were. And I was like, “Right on.” And then soon after that — Chris had joined us a few months prior to that and so we all had time in our schedule to start thinking about this. So we got to work.

Brent: So Chris, you weren't hired specifically for this. You just found yourself on project pretty soon after.

Chris: Yes, yeah. I got brought on to do some work on the product sites and some of our first runs in our apps. And I got asked by Grayson to take part in this, mainly as kind of like a help for Troy cause he was the one that was going to build it and it quickly turned into just me for about a year, until we brought Greg on.

Ken: For the front end piece, yeah.

Brent: Ken, some subset of features has to be done first. So how did you choose those?

Ken: In an app as big as OmniFocus is, there are obviously a lot things to choose between. But what we did is we looked back at what we had brought to the iPhone first. And so in the very first version of OmniFocus for the iPhone we had an inbox. We had project lists and their contents. We had context lists, and their contents. And of course contexts now have been replaced by tags, but that gave us the bare bones sort of structure of what we needed to have in place for us to feel like yes, this is actually OmniFocus running on the web.

It's not everything, of course, that we would love to see in OmniFocus. A bunch of the features like forecast and custom perspectives and so on, are things that we developed over time. This is just the starting gate, not the end point. So. We'll continue to build more features into this over time.

But we thought what is the minimum amount that we would need to give somebody for them to feel like, “Okay yes. I'm able to use my stuff now.” And they wouldn't have to wait for us to finish building all the rest before we continued the process. That's where we've started.

And similarly we... when it came to deciding what platforms we were going to build this for, we decided specifically to target those Windows customers who have been sending in a bulk of these requests. Not that there isn't value in also making this work on other platforms, on other mobile platforms for example. But the first place is to go to those big Windows screens on desktops.

Brent: Makes sense. Grayson, you mentioned that there were a lot of requests for this in OmniBugZapper. Did those requests have enough data to kind of help in the decision process?

Grayson: I think so. In that regard it was really great that we had that many requests. Because we had a lot of feedback within each request to figure out what direction we needed to go. And so the UX team started developing a few personas around some of the use cases that were in that bug. And based off of Ken and the executive team's sort of direction, we decided to zero in on the Windows use case, primarily. And so after doing that initial persona work and research it made it pretty easy to just move in that direction.

Brent: So what about the design for the actual app? I noticed that it looks an awful lot like OmniFocus 3 for Mac. Was that a decision from the beginning? Or various things tried?

Grayson: There were a few different mock ups that went down two roads. The first road was something that was kind of more all around in terms of friendly for mobile and friendly for desktop. As we kept talking about the use case we wanted to fill and create the ideal OmniFocus version one experience for — it became really easy just to be like okay. Now we're gonna go down... this is a great idea. This initial version of mobile and desktop. But we're gonna focus just on the desktop.

Grayson: And we have a great desktop client already so the UX team felt like it was pretty easy just to take a lot of those UX decisions that we had already made for the Mac, bring those over to the web. I think it was just a natural sort of transition from the Mac to the web.

Brent: The web is different enough, though, are we finding limitations or mismatches or—

Grayson: Yes. For sure. And that's been the real challenge. Every week there's another challenge. Today specifically was keyboard shortcuts. I'm sure tomorrow will be something else. The day before was inspector issues that Chris brought to me. So yeah.

And so it's been challenging because none of us on the UX team have created a product like this before, and so it's required a lot of research and exploration and talking with the developers and realizing what the limitations of the web are. Some of the things we just don't think about and we miss. Thankfully Chris has been here.

Brent: Well let's switch gears from design and front end and go all the way to the back end, which is where Mike comes in. Tell us about the back end. What's going on there? Big mystery to me.

Mike: Yeah, so there's really three components to make up the back end. The first is really what the user is connecting to when they're working inside the browser, is the instances of OmniFocus itself. So these are UI-less applications that are exposed through our system that the user can connect and work with their database.

Brent: So when you say UI-less applications, are we actually talking about the Objective-C and Swift code that makes up OmniFocus for Mac and iOS, but running in the cloud somewhere?

Mike: Exactly, yeah.

Brent: Cause who would do that? That's crazy.

Mike: It's totally crazy. But we've done it.

Brent: That's so Omni.

Mike: When you're running against this instance of OmniFocus, it's just like you're using your desktop application.

Brent: No kidding. I just find that amazing. Alright. So it's literally like the same shared code. That's—

Mike: It is, yeah. And we have multiple API hosts, which is the second component that these are running on. So that allows us to shut down one API host and update it while the users are still able to use the other one to run their database against.

Brent: So what else is going on then, in the back end?

Mike: Well the third component is the coordinator service. So this initiates and guides the user to their instance of OmniFocus.

Brent: Now this at least is written in Go or Python or something.

Mike: Yep, this is a Python application. We have a great experience with Python here at The Omni Group, so that's what we're using for our coordinator service.

Brent: So the instances, are they sandboxed in some way?

Mike: They are. So we dynamically provision users on the local API host to run these application instances. So that way, one user isn't going to be able to connect to another user's database. Also the instances themselves, being just like the application, that is where all the decryption of your OmniFocus database happens.

Brent: So yeah, let's talk some more about the encryption. How does this go on? So if you're just using OmniFocus for Mac or iOS you can have end-to-end encryption. The data's only decrypted in your client app that you're running. But if you're using it over the web, how is that different?

Mike: Yeah, so we don't keep your decryption keys or anything like that. That's solely your information. So when you log in, you pass that key to the instance itself, and it uses that to decrypt your database so that you're able to access your data.

Brent: And when you sign out it tosses the key, I guess? Or destroys the instance or...

Mike: Yeah, we're actually still working on that. But yeah, it will shut down your instance so that nobody else can connect to it and look at what you have.

Ken: That instance, we never actually store your key on in permanent storage. We store it in that instance's RAM memory, so when that process stops then that key is thrown away. What is left, though, as part of the syncing process and running with your data, we synced that data to that instance's local file store and there it was loaded into a database. And that database is necessarily unencrypted so. It was decrypted during that process and the clean up I think there is what Mike is talking about, we still have to clean up behind the session when it goes away. So it wouldn't be able to see any more new data at that point, but until we clean up the old data, of course, somebody scanning that disc could find it.

Brent: So naturally we won't ship until bugs like that are all fixed. And by the time... We're recording this in early September. It may be done even by the time you're hearing this.

Ken: That piece of it, I hope will be. That's certainly one of the pieces that we're waiting for before we go onto a wider set of testers.

Brent: So are the API hosts running on Macs? I assume they must be because they're running Objective-C and Swift code. Or have we done something crazy to get this up on Linux, or.. ?

Mike: Nope, they're running on Apple hardware and macOS.

Brent: Wow. We're such a Mac company that our web app is really a Mac app running on a Mac.

Ken: We did, you know, contemplate running it on iPads or something instead, since that also has the same—

Brent: That would be cool. I'd love to just see like a wall of iPads running OmniFocus for the Web.

So one thing that occurs to me, since this is all running shared OmniFocus code, that it's possible that someone fixes a bug in that code and it would go to the web app before it might make it out to the iOS or Mac app. That is, assuming you're using, what do they call it? Continuous integration?

Mike: Yeah so we really subscribe to what the rest of the tech industry subscribes to. And that's continuous integration and continuous delivery. That means that when we check in code, we have processes that build, test, and then push it out to our internal repo. So when that build of the OmniFocus instance is pushed the internal repo, it then is made available to the API host and users that are logging in will get the latest and greatest bug fix and feature rich application.

Brent: That's pretty cool.

Ken: We assume though, is that at some point we're going to want to have slightly more stable track than just whatever somebody happened to check in last.

Brent: Oh sure.

Ken: Even if it passed the automated test or something. So we'll have a test track and a release track sort of like we do with our Mac apps. People can hop onto the test track and use the latest stuff and help us find new bugs that we just introduced, or they can sit back and wait for other people to help to find those bugs and know that they're using something that's been more tested, a little more stable.

Brent: So how is the front end code published?

Mike: We're using CDN to publish our website.

Ken: Content Delivery Network, for those...

Mike: Exactly. So we have some automated processes that when we feel like a build is good to go we then push it out. The nice thing about a CDN is that the users won't see any interruption in service when we do those updates.

Brent: How does that work? I've never used a CDN.

Mike: I'm gonna try to give you the 30 second version here. So there's a main place that we store the code at. And then there are edge locations around the world that are set to time out, and when they time out, on the next request for that code it goes out to the base storage and pulls it out and that way it's cached around the world. So somebody in Australia doesn't have to wait 30 seconds to get the page to load. It's loaded into that cache automatically and they're able to get the website immediately.

Brent: So how's the front end written, Chris?

Chris: It's written all in JavaScript. It's mainly written in React, which is a library written by our friends across the street. Facebook...

Brent: Listeners, you should know we have one Facebook building and another Facebook building. One next door and one across the street.

Grayson: We're surrounded.

Brent: We are surrounded by Facebook, yeah. So if we have any problems you just walk across the street and find the people that... yell at them until they fix it?

Chris: I haven't had any yet.

Brent: Oh okay.

Chris: But I'm assuming if the time comes, I might have to...

Brent: So why did we choose React?

Chris: I mainly chose that because it was a small learning curve. I have been writing JavaScript for a number of years and it's just a regular JavaScript. So it's not too difficult to pick up. And it had a really cool feature that I liked called the Virtual DOM.

Brent: I know about the real DOM. That's a thing the browser creates with the nodes and stuff in the page, right?

Chris: Yeah and re-rendering that is very costly, computationally.

Brent: [Re-building]? the whole DOM?

Chris: Yeah. So what React does is that every time you make a change in your data it renders in the Virtual DOM a complete re-render of what would be in the DOM, and then it does a diff between what was in your Virtual DOM and what is currently in there. And then it only re-renders to the page what had changed.

Brent: Oh, that sounds pretty clever.

Chris: So every little component, only the ones that are edited or you get data sent back. That's all that changes as opposed to having the entire page re-render every time. Especially when you get updates from your other devices. You don't want it to re-render every time.

Brent: Right, sure.

Ken: Speaking as someone who once wrote web browser I sort of feel like that document object model... that differencing logic should happen in the browser itself. But it doesn't happen in a lot of browsers. And so we want to support a lot of browsers. It makes sense to do this in a way that will load quickly in all of them and update quickly.

Brent: I've heard people often say that the best thing about React is how... I might be using the wrong terms... how when some bit of data changes, some piece of the UI updates not only efficiently, but it knows which pieces. How would you say what I'm saying, only in the proper way?

Chris: That's almost just about it.

Brent: To do all that by hand is difficult, where as React makes that a lot simpler. Right?

Chris: Right. Yeah. Mainly just all the work they put into the Virtual Dom. Making that code diff. I'm not exactly sure how it works internally. I just know it works and works well.

Brent: So one thing I'm curious about, and I hope the answer is yes. So, say I've got two computers in front of me. One is running OmniFocus 3 for Mac. One's running OmniFocus for the web. And they're both going and I add a new item on OmniFocus 3 for the Mac, swivel in my chair, look at OmniFocus 3 for the web. Will I see that new item there?

Chris: Yes.

Brent: That's amazing. How?

Ken: When we were working on OmniFocus for Mac and iOS and building up the syncing system... a lot happened in these intervening years from 2010 'til now, where we first started building out our own syncing service, as Apple moved away from .Mac and MobileMe and letting you sync arbitrary data to their servers. When we built our syncing service we wanted to have changes that were made on one device push immediately to the other devices.

So we ended up building our own push notification service because Apple's service... they had one that was available, but you could only use it for apps that were in the App Store. And our app is in the App Store, of course, but it was also distributed outside the App Store, and at that time the copy that was outside the App Store couldn't use that service. Since then, they've relaxed it and we are now able to use their service.

But anyways. We had this infrastructure lying around that we had built for maintaining these live connections, basically, to all these instances on OmniFocus and when a change would happen we would send a notification saying, “Hey, a change happened, you should sync. Go look for it.” And so OmniFocus... these new server instances that we've built, we're running quite a few on the same host but they still have a lot of that same underlying logic and one of the pieces is they're able to receive these push notifications from our system.

So they get notified when you make a change on iOS or on another copy of OmniFocus for the Web. Say if you're running it on two web browsers and it will push the change down to the sync service and then send a notification saying your other devices should sync then.

And then that API instance maintains a connection to your web browser to let it know, “Hey. There's some changes. Maybe you should refresh.”

Brent: Networking.

Ken: It's sort of magic when you see it in action. You just make a change and suddenly it is on the other devices right there next to you.

Brent: And working on the front end, have there been any specific challenges like, oh I didn't know that whatever this one thing would be super difficult? Or has it been more of a like... you know, just every day there are things.

Chris: Yeah, every day there seems to be something we run into. I remember when we were first implementing drag and drop. Realizing just how not-fantastic the web APIs for drag and drop are. That was a bit of a struggle. But yeah, it just seems like there's always something. But we manage to tackle it.

Brent: That's software for you. There's always something. So at the time we're recording this, it's September 5th here. You're listening to this in October. We're up to a few dozen testers Ken?

Grayson: Forty-one.

Brent: Forty-one. So and by the time this goes out it'll probably be a few hundred, maybe?

Grayson: I sure hope so.

Ken: I hope so too.

Ken: We have how many thousand on the waiting list?

Grayson: I thought we had 4,000 or maybe—

Ken: Sounds about right.

Brent: So what are we learning by just the slow addition of testers? Other than, I mean we're finding bugs, obviously.

Grayson: That we have a lot more work to do.

Ken: Our first goal for what we wanted to learn out of this small set of testers was, could we keep the servers up? Could we make sure that when people were logged in and signed in, as we updated the software they would get the new versions of the software without inconsistencies. And we did run into some bugs there and we've flushed those out. There may still be more, I don't know. But at least some of them are fixed now.

And started to learn how much load on one of these server hosts, one of these Macs... How many of these processes can it run at once? How many copies of OmniFocus can it run before it falls over? Because we're not going to run all these thousands of people on a single Mac. That's impractical.

So right now we have... as Mike had alluded to... we have two Macs, I believe that are set up. One that's kind of a fail over for the other, but perhaps by the time you hear this we'll have to have more, based on how many people we're inviting in.

Brent: So the question is scaling, really. How many instances per machine?

Ken: And that helps us to understand how much it costs to actually run this service, and so how much we will have to charge to keep the service profitable.

Grayson: And from a UX perspective, from this first group, nobody ran away screaming. [crosstalk 00:24:31] I had some mild concerns about some of the decisions we made. I think we're on the right track.

Brent: What will pricing be like? We don't know, I guess, until we know more about scaling I suppose.

Ken: Yeah, we know that it won't cost as much as a Mac costs. Although there's still some hosting costs involved and so on. You know, but each person is sharing a Mac with a number of other people, so effectively they're all renting a Mac together and so it really boils down to how many people share a Mac at once.

Of course there are other costs involved in developing this. Running the service, the networking costs, and everything else. But that is certainly one of the drivers that was an unknown until we start to get enough people on there to make one of these Macs fall over.

Brent: When you say renting a Mac, that kind of implies that this might be a subscription based service?

Ken: Yeah, and of course that is a new thing for us. We have built software that runs on people's devices. Their own devices. They're buying our software. We, of course, continue to update the software. But when somebody purchases a copy of OmniFocus today, they can run that copy for as long as they keep their hardware running. And some people are still running OmniFocus 1 from ten years ago. Or some people are running OmniWeb 1 from more like 20 years ago...

Brent: On their NeXT machine. That's cool.

Ken: Exactly. That's a different scenario than what we're talking about now. This is the first time where we actually have to run some hardware to let you use the app at all. That costs us money to run every month, and we need some incoming money every month too to keep that going.

So we'll be doing some sort of a subscription service. We have not yet determined that pricing because we have not yet determined the cost. Our thought is, though, that if we're charging somebody money every month, let's go ahead and throw in the one time costs involved in buying the app as well. So OmniFocus for the Web is not a stand alone product. It's not currently envisioned that way. It's a product that you use with OmniFocus on your Mac or OmniFocus on your iPhone or iPad.

But if you're paying us some amount of money every month it might feel weird to then have to buy it again for a different platform. So our thought is we would just do a universal OmniFocus subscription price, and if you subscribe you get OmniFocus everywhere.

Brent: That's pretty cool.

Ken: We hope so. We think so.

Brent: And in the end — we don't know the price yet. But I imagine it will be in the ballpark of other kinds of productivity app subscriptions?

Ken: I sure hope so. If it's too far out of the ballpark, then I don't think we'll get many sales.

Brent: So Grayson, what's the future look like for OmniFocus for the Web?

Grayson: I'm hoping our listeners tell us. Like Ken mentioned earlier in the episode, Forecast, we definitely want to get that in there at some point. Custom perspectives. Seen a few calls for a Review in Slack. And just making it better and faster and refining things, right now. But I'm sure we'll get a clearer picture of where we're going by the time this episode is published.

Brent: Well it seems like we're at the end of a long road, hurrying up to get to the very beginning.

Grayson: Totally. I was really struck with the day we went into a test and the feedback started coming. I'm like oh... It's basically like starting over. Here we go.

Brent: Alright, we'll close this off. Well thanks, Ken. Thanks, Grayson. Thanks, Chris. Thanks, Mike. Ken. Where can people find you on the web?

Ken: They can find me on our website, OmniGroup.com and also on Twitter at @KCase.

Brent: Mike, where can people find you on the web?

Mike: People can reach out to me at LinkedIn. I'm always game to talk tech.

Brent: I'd also like to thank our intrepid producer, Mark Boszko, who had an extra big panel to deal with today. Say hello, Mark.

Mark Boszko: Hello Mark!

Brent: And especially I want to thank you for listening. Thank you! Music.

[MUSIC PLAYS]