In our First Line of Code series, Commit co-founder Beier Cai talks to prominent tech founders about the early days of their startup.Ivan Yuen is the founding CTO and Chief Strategy Officer at Wattpad, a social storytelling platform that enables new and established writers to share their stories. Founded in 2006, the community connects over 80 million monthly readers and writers around the world.
Commit co-founder Beier Cai sat down with Ivan to talk about the early days of Wattpad and what it was like to write the first lines of code.
Beier Cai: How did you get into coding?
Ivan Yuen: I loved playing with computers when I was little. I was the kind of kid that when there was extra time at school, I would beg the teacher for time with the computer. There weren’t a lot of resources available either. It was like, here’s a computer, here’s a bunch of magazines and manuals—go figure it out.
Beier: What was the first software or piece of technology you built?
Ivan: I built lots of things for fun, including Wattpad. I basically hacked Wattpad together over a weekend, and that became the start of it. That code evolved to what it is today. That’s one of the things that I took away from the experience: software is never finished. You start something and you live with it forever. I’m of the mindset that if there’s something that’s poorly designed, something that’s not working well, you can improve it over time.
I had to resist the temptation to throw everything out and start from scratch. That’s very tempting to do. Especially now we’ve carried on with 10 years of technical debt, but we’ve continued to rework components and rewrite parts of it. In my mind it’s very, very dangerous to take the whole thing and throw it out and start from scratch, because you’re throwing out 10 years of bug fixes, you’re throwing out 10 years of business logic and nuance that may not be captured anywhere.
Whatever decision you make early on, you kind of live with it for the rest of your life, but you need to recognize that it’s an ever-evolving thing. You don’t have to get it right the first time.
Beier: A lot of engineers, especially in the early stages, struggle with the fact that there are so many new technologies out there. As someone trying to build a new product from scratch, how did you choose between tested and true technology and shiny modern tech?
Ivan: When I started, I decided I wanted to build on PHP because that was the cool, trendy thing to do. It was less about whether it was the right technology. The question was ‘What do I want to learn?’ That’s what I based my decision on.
Now, is that the right thing to do? I don’t know. I don’t think there’s a right or wrong answer, but that worked for me because I was able to keep learning. I think that also helps for forming an early engineer-first environment, if you want to attract like-minded folks that are also interested in playing around with the latest stuff.
I think it can be good to go with the latest and greatest, but I probably wouldn’t advocate for always picking the newest, shiniest things. You don’t need to reinvent the wheel across your entire technology stack, right? Choose the parts that are interesting, that might give you an advantage versus other things that are proven. I don’t know if I personally had the balance right, but that’s what I would advocate for folks who are early in the process.
Beier: Can you share some of the guiding principles you have in terms of when is the right time for you or your team to adopt new technology?
Ivan: When it comes to adopting new technology, for us anyway, it was always the need first.
It’s not like, ‘Here’s a technology, what can we do with it?’ It’s ‘We have this problem, let’s deal with the problem first.’ In terms of how to solve that problem, I would say it’s always been bottom-up. It’s very collaborative, where the team gets together and discusses the different ways of doing it.
Maybe it’s not clear which way to go, so let’s experiment with both or three different options and then come back and share that knowledge. I would say that’s very much the culture. It’s never been like, ‘I’m the CTO, I’m going to go in and say do this, this and this.’ I don’t think that’s ever worked for us—it’s not something we want to do.
Beier: So essentially it’s guided by the problem you are trying to solve, and then you experiment. Have you tried to measure technical debt and how it affects your team performance?
Ivan: In my previous life I’ve done things like that, like when there’s a defect you can say okay, what was the cause of that defect? Was it because of a poorly documented area of the code or lack of testing?
What we do now is based on engineering sentiment. On a regular basis we survey the engineers and ask what are the parts we should address next, to get rid of legacy code or deal with technical debt. We get a sense of what areas are top-of-mind for engineers, that they’re dealing with on a daily basis, as opposed to something that might be bad but, even if we improved it we’re not going to see any benefit.
Beier: What was a major technical milestone you accomplished at Wattpad? Why did you choose to do it, and what was the result?
Ivan: It sounds mundane now, because it’s the default, but we adopted cloud technology very early. We started using AWS when it was in beta, in late 2009. It was EC2 and EBS.
The tools were very primitive, there weren’t a lot of other people using it, so it was hard to learn because there wasn’t a lot of knowledge-sharing, but we did it because it was easier for us to measure scale. We saw that it was going to be something that would help us scale in the future.
Even though at the time it was labelled ‘beta’—anyone that is relying on their system to be mission-critical would not use it—as an early-stage startup we were able to take those kinds of risks. We just decided to jump in. We didn’t know what their technology roadmap was, whether they were going to serve us in the future, but we thought, this is something that will help us right now in terms of scaling, and will also give us skills and knowledge to learn early on.
We literally went from a managed hosting service to the cloud—I want to say over two weekends, something like that. And to see it work, to see everything running there and to see that seamless transition—it was effectively zero downtime in transitioning the server load over, and everything was invisible to the users—I was really happy to see that.
Beier: Do you remember the moment where you and your team made your back end system five times or even ten times more scalable? What did you change to make it happen?
Ivan: There were a couple things. In the early days, the bottleneck tended to be in the data system. All the way up there in MySQL or the Memcached layer. You don’t want to overprovision anything, because it’s very expensive. So we would monitor it and say, ‘Okay, it’s getting to 80 or 85 percent capacity during peak, let’s add more capacity.’ I think we learned early on that by adding more capacity, we always saw our usage take a step function because of that increased capacity.
The database at our peak might be at 80 percent, and we thought, there’s still slack, we should be able to handle more traffic. But once we’d add capacity, we would see peak volumes go up by 30 or 40 percent. So we realized that if we could overprovision a little bit, it actually facilitates growth, because everything will be faster. It can handle more volume.
Another example is caching technology. We’ve used Memcache and Redis for a number of years now, but we used them out of the box, we just assumed that they worked. We never put a lot of effort into digging into how those systems work and the different optimizations that can help us get the most out of them. The lack of effort to understand and tweak the system led to many performance issues and bottlenecks. For example, we learned that certain key values were not being properly compressed, which led to memory over usage, fragmentation and ultimately slow performance.
It’s easy to look at those systems and think this is just a key-value store, we just put stuff in and we’ll get stuff out. But there are so many things that can impact the performance of it, so it depends how you use it.
I think the lesson is whatever system we end up using, we better know it damn well. Don’t take it for granted that it’s just going to work. Be prepared to invest time to understand it, to tweak it, to make it really work for you, because your use case is going to be different from everyone else’s.
Beier: The first few hires are really important for a startup. If you were to start a new company today, what qualities would you look for?
Ivan: Number one is people who are scrappy, people who are willing to do whatever it takes. They don’t have expectations, like ‘I’m going to work on this technology and this technology only,’ or, ‘I’m going to build this.’ My experience has been that things are going to change, so we need people who are adaptable and willing to take on risks.
The second thing I look for is people that I enjoy working with. What worked for us was hiring a lot from people I’ve worked with before, and people I know can talk on the same wavelength. I don’t want to be spending a ton of time explaining a lot of the whys.
In the early days, I didn’t take into account the specific technology or the experience with the type of technologies that we planned to work with, because, again, I’m looking more for flexibility and the ability to learn.
Beier: Would you specifically hire senior engineers as your first and second engineers, or are you open to any kind of seniority? I see a lot of startups these days who say their first engineer has to be senior.
Ivan: If I were to start today, I’d probably go with someone less senior. I’d be okay to work with someone less experienced, as long as those other criteria apply—that they’re curious and they’re willing to roll up their sleeves. Maybe because I have a technical background and I can complement some of that. If you’re hiring and you don’t have a technical background at all, and you really rely on someone to make those technology decisions, then I can see why you would want someone more experienced.
Beier: How did you create a sense of ownership or a sense of urgency in the very early days among your team?
Ivan: I think what helped instil a sense of urgency is that me and Allen, my co-founder, we just wanted to dig into the problem. Any time of day, it didn’t matter how deep, it was like, ‘Okay, here’s the problem, let’s go in and fix it.’
I think this was made clear to that early group of people—whether it’s the first engineer or the first customer support—if there’s a problem, if there’s a system outage, I’ll have my laptop in a backpack. I’m going to pull it out and see what the problem is. Lead by example. If there’s a fire, I’ll charge in first. As an engineer on the team, if you see your leaders going in, you feel compelled to follow along.
When it comes to the technical side, I’ll ask questions, like, ‘Have you thought about this and this and this?’ And, ‘I think this is a problem. Do you agree? if not, tell me why.’ Ask those questions not with the intent taking away their ownership or accountability, that’s not the point. Hopefully through that conversation we can get to a better answer.
Beier: Can you share with us one of the biggest technical project failures that you experienced, and how you came out stronger?
Ivan: There are some things that, in hindsight, we would’ve decided differently, but I think they made sense at the time.
Wattpad has a pretty data-driven culture. Data collection, and being able to make an informed decision based on that, is very important. We realized early on that the analytics systems that were available at the time did not meet our needs. At the time maybe the gold standard—and it’s still popular today—is Google Analytics, but it was very limited in terms of how it tracked users.
We wanted to track a user through their lifecycle. To be able to capture information down to when they started reading, how quickly were they reading, where did they start reading, where did they stop reading. There was nothing off the shelf available to do this six, seven years ago. So we thought, let’s build it.
Initially, building it wasn’t too difficult. But the mistake we made was underestimating the effort to maintain that system and to update it based on our needs, to keep it state-of-the-art. To make sure that we can get value over a long period of time, we had to invest a lot of effort and continue to build and maintain that. We only looked at the upfront costs.
Nowadays if we wanted the same thing, there’d be a number of off-the-shelf solutions, so we wouldn’t have to build it. I think the lesson here is that if it’s something that’s readily available elsewhere, then we lean towards buying, because we want to build things that are unique differentiators for us. If we were to build then we’d have to maintain it. It adds to the operational burden over time, which is something that is going to continue to slow us down as we get bigger and bigger.
Beier: Technical leaders are responsible for building technology, but there’s actually a lot of other things going on. One of our engineers from Commit became a CTO at a startup, and he asked me about how to manage a board meeting. What do you do, what do you say? What are some of the lessons you’ve learned about responsibilities outside of your day-to-day role as tech leader, and how have you learned them?
Ivan: One thing I started doing just three or four years ago is connecting with a small group of tech founders. We get together and talk about our problems, these type of questions we’ve been talking about now: how to manage a team, how to manage strategic trade-offs, how to grow personally, how to talk to a board of directors. Things like that.
It takes time to build trust in a small group of people, but being able to share any problems, to speak honestly about issues you’re dealing with, and seek specific advice… A peer group is something that I did not have early on in my career. It’s something that I did not actively seek out or think was that useful. But having had one for the last few years, I would say it’s immensely valuable.
Beier: Starting your own business is not an easy choice. You could’ve joined Amazon, Facebook or Google and found a comfortable job. How did you make the decision to jump into the startup world and start your own company?
Ivan: I went through Waterloo’s co-op program, and throughout my co-op experience I always worked with big companies—the IBMs and the AMDs—and I thought that’s what I would do on graduation. But on my last work term I happened to join a software startup and that was it. I was like, okay, I can’t do anything else, I have to be in the startup environment. It was just more dynamic, more exciting, and more rewarding. From then on that’s all I ever did. So, for me, it was an easy decision.
Beier: From a financial perspective, did you plan out how it would work?
Ivan: There was a realization that it’s very different joining a startup versus starting your own company. The financial security, the financial risk, is different. When I decided to start Wattpad, it was at a point in my life where I could take that risk. I didn’t have a mortgage. I was married and my wife was working full-time. I thought I could go maybe a year, maybe a year and a half with no income. That reduced the financial stress. Once you go and take out a mortgage, your tolerance for that is lower. Once you have a couple of kids, your tolerance is lower. At the time I recognized, you know, this is as good a time as any and it’s only going to get harder from here.