ABOUT JEREMY KING

Jeremy King is Senior Vice President of Engineering at Pinterest, where he leads the company’s technical direction and oversees the entire Engineering team building deeply technical products, platforms and machine learning systems.

Previously, he was the CTO of Walmart, where he led the digital transformation effort of the company including customer technology, merchant technology and supply chain technology that covered all Walmart U.S. stores and eCommerce. Prior to that, King was Executive Vice President of technology at LiveOps, and Vice President of engineering and software development at eBay.

He holds a bachelor’s degree in information technology from San Jose State University and is an advisory board member for the CTO Forum, an organization that brings together senior leaders across the technology industry to collaborate on key issues and accelerate innovation across organizations.

"What I typically tell CTOs and heads of data is that your data probably is biased already. So you need to go back, check your data sets, and go reset the data from the very beginning."

<cite>- Jeremy King    </cite>

Check out our friends and sponsor, Jellyfish. Jellyfish helps you align engineering work with business priorities and enables you to make better strategic decisions.

Learn more at Jellyfish.co/elc

SHOW NOTES

• Building inclusive products starts by having diverse data sets (2:13)
• Why your data is probably biased already (4:33)
• Where to start with building inclusive products (6:15)
• Three principles to build inclusive products (9:16)
• How Pinterest disrupts entrenched patterns of thinking & balances innovation and action (11:17)
• How to decide which experiments to implement (14:36)
• Why ROI should not be the only metric of effectiveness (16:11)
• How to filter decisions through your company’s mission (20:24)
• How AR aligns with Pinterest’s mission & allows “Pinners” to explore & experiment with different identities (23:08)
• Covid’s impact on retaining talent (24:40)
• How “investing in rest” & cultivating work-life balance can increase productivity (26:49)
• How to schedule a day off for your entire engineering org (29:57)
• Upcoming Industry Challenges: building in serendipity, onboarding in a remote-first workplace & the logistics of workplace flexibility (31:48)
• Rapid-Fire Questions (38:06)
• Takeaways (43:56)

LINKS AND RESOURCES

• Inclusive Search and Recommendations - Nadia Fawaz’s talk on How Inclusive Search, & AI works at Pinterest

TRANSCRIPT

Patrick Gallagher: Jeremy, we are so excited to have you on the podcast. Thanks so much for joining us and officially welcome to the engineering leadership podcast.

Jeremy King: Wonderful. Thanks for having me.

Patrick Gallagher: We wanted to talk about building inclusive products and how to make that easy and actionable for engineering teams. And as Jerry and I were sort of preparing for this conversation, it became really clear to us that there's not a whole lot of insight behind the scenes as to what this looks like within engineering teams.

Can you share with us about the process of building inclusive products and what that's looked like at Pinterest?

And then maybe we can start to deconstruct some of the mechanics or frameworks within that conversation. So what does that process look like of building inclusive products at Pinterest?

Building inclusive products starts by having diverse data sets

Jeremy King: Yeah. Well, I think the first point... it's going to be maybe a bit disappointing. It's not easy. So there is no easy insight on it.

And I think when we think about Pinterest... the nice part, I've been here two and a half years, but you know, Ben and Evan had this mission to build a platform that was beautiful and that people loved.

Both Ben and Evan are designers and architects and you should see their collections are just unbelievable. And so they really did want a product that people thought was beautiful.

And as people come to Pinterest for inspiration, just like any startup when you've got early adopters. Oftentimes the early adopters are typically one demographic... often it's a bunch of geeks, you know. In our case, it happened to be, middle-America crafts and that sort of thing.

And what we found like, as we began to get feedback from our users. We got some quotes like, "Hey, I don't want to have to put, ‘black’ in front of my query or ‘Asian’ in front of my query."

We knew with both our I&D teams and our engineering teams that we really had to get to work.

The way you have to fix this is actually to step way back.

Once data has been built into your models, then trying to unwind that sort of learning is really tough, right? You have to like over-inject and boost up features. And that sort of thing will work for a little while, but really what you have to do is go back and retrain.

And so we had to go... let's step way back and make sure that the data that we fed the engine, was diverse from the outset.

We built a product called skin tone, ranges. If you do a search for like lipstick, it'll ask you, right on there, “what skin tone would you like to see?” And let you pick that. And you can imagine the technology behind that is extremely complex. Not only from a, just an engineering standpoint. But the computer vision side of that is incredibly hard. Just detecting skin tones, we've had to go through a series... It's been like four iterations of how to get that right.

Back to your question, it's like, “is there easy points?”

What I typically tell CTOs and heads of data is that your data probably is biased already. So you need to go back and check your data sets, and go reset the data from the very beginning. And that's... that's really quote unquote “the easy part” but it's actually harder than it sounds.

Why your data is probably biased already

Patrick Gallagher: My immediate follow-up was like, well... how do you check the datasets? like, how does that conversation typically look? Or who are you engaging with to, to do that sort of assessment piece?

Jeremy King: Yeah. Great question. So we of course used our I and D teams. But we also hired some wonderful people. We have, the woman that runs our inclusive engineering team Nadia Fawaz. And then we have a head of product named Annie Ta and then the original person from I and D was a woman named Candice Morgan who's now part of Google. And the three of them together sort of built the whole ecosystem. Like here's what we have to go do it.

Then you have to go out and literally build a diverse data set. In the cases where we didn't have it. We had to go get it.

And it wasn't so much that we didn't have it. We're just way biased on one side than the other. So you have to sort of downplay the others. And then you just continue to add features on top of it.

We knew that skin tone ranges was the number one feature that people know. And then we just launched hair pattern, which you can imagine is extremely complex. Curly hair, straight hair, bald, all the above, trying to identify that in billions of pictures and to make sure you're sourcing that is a really fun technical challenge. So Nadia and Annie have the ball on this.

The other thing we do is... a new user that comes in through the front door... what do you show them? And it's like, how do I show diverse feeds from the very beginning when somebody comes in and does a search and say, “Hey, I'm interested in beauty products” or “I'm interested in fashion.”

What do you actually show there?

And a lot of our growth is coming outside the US as well. So we not only have to build diverse content from a skin tone and a nationality range, but we also have to think about it globally as well.

Where to start with building inclusive products

Patrick Gallagher: I'm curious about the like early-onset conversations, like at the beginning, when everybody's coming together as a team... to start the conversation of strategically, how do we begin this process of building inclusive products? I was wondering if you could tap us into some of those early conversations...

What were some of those early conversations like with the team?

Jeremy King: Yeah, first thing was, we were really listening to the community, right? And so you get this feedback coming back from your user base. And again, it might be a relatively quiet side of the user base because you're thinking about a new diverse set of people that are starting to use your product. You may have to be listening very carefully for that. So number one was just looking at the diverse feedback from our pinners.

The second thing was really figuring out what we can do about it. I got a download from Nadia and Candice when I first joined about how to decide. And you can imagine the challenges... these are problems that it's not something that an engineer can fix in a week. This is a really hard problem to work on. And most companies have tried it as you've seen... have failed computer vision, face detection and all kinds of issues with, especially darker skin tones initially like in lighting conditions. And these are like, these are advanced, PhD level, problems to solve!

We hired Nadia and Annie to focus on this, pick a few projects. And then, frankly, we let them experiment. The good news is Pinterest is built on a on a great experimentation dashboard and a platform. And so we were able to let them run with it for a while. And just, frankly see how it was working.

And as you can imagine, our early attempts were pretty bad. Nadia does a wonderful talk about this. Initially we went after face detection and there's all... it's rife with issues. Even just turning, how do you detect that a face faces even there. and especially when the pandemic hit, you know, you got everybody wearing masks and all kinds of different color masks and things like that.And so it was, uh, a really hard problems.

So we abandoned that completely. And started down a new approach just to detect skin. Like, can I identify that there's actually skin in this image?

And it's a fascinating computer science problem. But it's really, giving them the permission to go work at it right. And dedicating some resources and letting them rock and roll.

So it's a really great story. It's about a... she does a.. about a two hour talk about how we went through the technology challenges which is great.

Patrick Gallagher: Oh, that's great! We'll definitely track that one down and add to our show notes.

Jerry, I've been asking a lot of questions. I want to open it up to you. What are you thinking about right now? What's on your mind?

Jerry Li: Yeah, I think this is a very good opening story to give people a really concrete example of what building inclusive products means with examples. That gives us the context to start a deeper conversation around.

I was wondering, Jeremy, if you can help us to deconstruct the process a little bit more.

Just to over-simplify the process, I can think of, well to build inclusive products, you first need to have inclusive ideas coming from the employees, or the team, coming from the customer. And then those voices are heard and amplified and included in the product building, design building process.

Three principles to build inclusive products

Jerry Li: I was wondering whether you have any thoughts in terms of the framework and processes people can follow?

Jeremy King: Yeah. First things first, I mean, Pinterest, as a company, this sort of, part of our mission to help everyone experience and build a life that they love. So when you think about everybody, when we think about that, so it's part of our mission anyway.

But to be fair, we really heard this from our user base. And I think, you know, this was many years ago that we started this effort. But I think many companies are growing around the globe have now heard this from their consumer base that, "Hey, we've got to build a much better product for our diverse user base."

So listening, listening is number two.

And then number three, I mentioned, which is really the key... is making sure that you get some experts in there to understand, if your data set is already biased. If it is already biased, which it probably is... Then to go address that by either filtering out the bias by leveling the playing field or to go buy or acquire some data that is more diverse.

Once you have a diverse set of data, then you can have your machine learning teams build on that data set to start.

Frankly, then the last part is test, get feedback! Right. And I think that's a fairly well-known, practice, but, make sure you've got a very diverse set of people who are vocal about whether the results are working.

And that's really work we've done. And like I said, we've done it over years of iterations at this point. So, it's really something that's hard to get right the first time.

Patrick Gallagher: One of the things we were talking about earlier was the conversations you have with teams to help them either get unstuck or to imagine new possibilities with the different technologies or products that you're working on.

And I think. Jerry and I, we were recently having a team conversation, and the subject of the conversation was about setting a longer-term vision. And admittedly, I was stuck going into that conversation. I was like, “oh man, like, I can't even imagine like... what I'm gonna have for breakfast tomorrow. Let alone, what we're doing is gonna look like five years from now!”

How Pinterest disrupts entrenched patterns of thinking & balances innovation and action

Patrick Gallagher: I know you had mentioned that you've had those conversations before to help people come to think about new possibilities or break certain patterns of thinking to think bigger.

So can you share a little bit about your approach to that and how you help people to think differently or disrupt some of those patterns of thinking?

Jeremy King: Yeah, this is a good conversation. This comes up a lot in the innovation discussion, “How do you help teams become more innovative?”

My brother, my twin brother is a product leader and always has been. And, he and I used to laugh that, I would complain about the technical challenges that we're trying to build. And then he would complain about the hard part of the product a role.

And, frankly, I feel like the product role is actually harder because the point of the product role is that you have to decide what to do. It's not how to do it. Once you've decided what to do...

And what I mean by that is there's all kinds of technical advances that are coming out. You'll have hack days that... we have wonderful new startups that come talk to us. We have some great scientists that help build our graph database.

And the problem is like, how do I apply this technology to a feature and then say, "Yes, that's, we're going to go after!"

And so it's... I feel like my role as not only the engineering leader is a very heavy influence on the product side. You show the art of the possible, and then you decide to go do it.

I don't want to say ideas are, I think the cliche is “there are a million ideas out there. The question is which idea do you go after?”

And so I'm a bias for action person. So I want somebody to say, "Hey, we've got this great new feature, this capability, and then let's go ship it. Let's actually get it out there."

Cause you can get products 80% of the way, and you can play around with it. But until it's actually in production, you don't know whether it's going to be useful or not.

So I push my teams to actually get it out there. And that's the startup way is, hack it into their product and see if it works. So I really am trying to get ideas to action and that's because I think there are a plethora of ideas out there. There's not a lot of people out there who can turn that idea into something.

Jerry Li: And that's very fascinating. Can you share a litte more about that example process? That idea that originated from the engineering team. And then this just go ahead and run an experiment... because you mentioned there's a really solid experimentation framework you already have?

Or does the idea come from product?

Jeremy King: Yeah. Luckily Pinterest was built from the ground up with experimentation built-in, like many of the digital only companies.

So my previous life at Walmart, you're sort of injecting experimentation platforms into platforms that weren't originally built for experimentation. And so it's much harder to do a test and learn a process on that. So number one, that Pinterest is lucky because we're largely digital.

But, Walmart, for example, did lots of physical experiments in stores where they rip out a bunch of registers and put some new ones in to see what that looked like. And really trying to build a physical iteration process that matched the digital and that included training and that sort of thing.

But again, it's like picking a few of them and like actually putting some meat behind it and seeing whether it's going to work.

There's a million hack day ideas that are sitting in the idea lab. You know that list somewhere that have never been realized... And it was really about having the courage to go try them

How To Decide Which Experiments to Implement

Patrick Gallagher: I had a, I guess a question about all of the different ideas. Is there like a certain framework or filter for how you and the team worked through... like, what's the “go no-go” or the experiment that you're taking a bet on? What's that decision making?

Jeremy King: Yeah. good question. We have sort of matured this last year. A guy on our team named Zartek took it upon himself to build essentially a feature... there's a few companies out there that do these sort of idea labs, startup hack day, idea labs, where you put in a series of ideas and the technology behind it. And then the company will vote it up and down. And the top 10 get funded or get prototyping dollars. And we sort of matured that this time

And we've got about... it's fun we have about 10 projects that are in the development. Several of them are internal productivity things versus things that pinners will see.But that's a well-known process. And usually it's done via a shark tank or whatever.

I was just talking with a good friend of mine, that was... they run like a shark tank process. And half the fun of that is just getting in front of the product teams to get your product in front of them with some feedback.

I highly encourage, if you don't have a mature hack day process for your company. You should put that in. And make it a fun event. And also, make sure that you fund... call it a half a dozen of those projects. Otherwise people will lose faith that they're just doing it for no real reason. Right.

And then celebrate the ones that go live. So I know a lot of companies that use ROI as well. And ROIs are sometimes... ROIs are game-able... that's all I'll say. So pick the ones that people have their gut feeling on. And then launch them in production.

Why ROI should not be the only metric of effectiveness

Jerry Li: Yeah. Speaking of ROIs, I am curious to learn that... a lot of features to make the product more inclusive, by nature, the way impact is measured is probably going to be very different from the mainstream way of doing things.

So how do you help to calculate or measure the ROI for those?

Jeremy King: Yeah. Great question. So the way Pinterest works, our experimentation platform has essentially a series of metrics across the board. Is it positive for new users? Is it positive for a cohort of these users? Positive from an ROI standpoint, from an ads platform.

And it's really interesting. I was, just telling the team this the other day, that rarely is any feature that we launch a hundred percent positive to every metric. And that's where the product management and product decision making comes in. Because you may say, "Hey, this is really good for new users, but it's not great for our users that are power users.”

Do you ship it? Or do you only ship it to new users and not ship it to your power users?

These kinds of decisions are extremely difficult to make with data alone because you have to see the data. and half of them, you would say, “If it's not a hundred percent, then just don't ship it!”

But that seems crazy. Right.

So again, we do have, because we have the experimentation platform, it gives us some freedom to launch things through very niche sets of users. It makes your system extremely complex when you do that. So, it's not just personalization, but it's, something else.

So, we try and make our bets on where we're going. And part of our values as a company. One of the reasons... People laugh a bit a bit at me when I, talk about Pinterest goals...

Our goal on Pinterest is not for you to stay on Pinterest for forever, right? I mean, like we call it... people call it “doom scrolling.”

We want you to actually be inspired to actually go out and do something like cook dinner, buy a new dress or, find out how to, you know, make a table for your home office. We want you to leave Pinterest to go actually out in the world and do things.

So it is maybe a little bit different, from a company culture standpoint, than others. But we try and put it around our values.

Patrick Gallagher: I think it's so interesting, especially in the world of increasing retention on a platform where it's “have eyeballs on our central platform as long as possible.”

Do you measure that? Like do you measure the conversion from idea to action? Is there a way to see that type of impact?

Jeremy King: Yeah, it's a matter of fact... this is one of the big things we're launching tomorrow is, we call it "takes" and I even did it this morning.

One of the creators that I follow does a whole bunch of breakfast stuff and he did this whole cheesy egg thing. And what you do is you get a video of like how he did it and then you film yourself doing it and saying, “I happened to add jalapeños and Siracha and a few other things on top of it” and say “this is my version of what they created”

and then I posted it back.

And so the whole point was to build a community around the recipe that I was following this morning.

And you can imagine that in recipes where people will build a vegan version of it, or a steak version instead of the chicken version or something like that. But you can also think about it as fashion or beauty or home or pretty much everything doing a creative project “and here's what mine looked like.”

Pinterest has been going on for years on what's called Pinterest fails. I don't know if you've ever seen that, even six, seven years ago...

Of course somebody would come out with this beautiful cake that they baked and it's all... And then somebody would try and attempt it. And of course it would come out as a disaster. Right?And so they would take pictures of like, “here's what it's supposed to look like. And here's what my came out like.”

And that's been a really famous meme as you do a search for Pinterest fails. You'll see all kinds of wonderful examples of people trying stuff, and then totally, making a disaster of it. So hopefully people will work their way through that and show their beautiful takes. But that's the idea.

And you know, what we're building is not an entertainment platform. It's... I'm a geek, so I will say something like utility. We want people to be helped. I want to go there and find out, "Hey, I need to go make cookies for my kids, Halloween party”

And then go out and do it. And then she'll share it with someone else. “Hey, I changed this a little bit and here's how it went."

And that can be a community versus just like a blind eye “like that” or “I'm following this person.” Right?

So we want it to be a community of people that are like-minded, if you know what I mean.

How to filter decisions through your company’s mission

Patrick Gallagher: I have to imagine... it's almost a tricky incentive.... Like how do you set up the incentives in the right way to create that?

How do you use like the mission of Pinterest within the decision-making of the different products and features and things that get released, in a way that's incentivizing those outcomes that you want people to achieve on the platform?

Jeremy King: Yeah, that's a good question. We've had a lot of debates about this internally and a lot of companies, especially in the social space, use "time on site" as their metric. We don't use that.

What we want is more... call it “sessions.” Like I would love for you to come and get inspired about what to make for dinner every night. And come and say, “Hey, I want some chicken recipe tomorrow. I want a soup recipe.” And come every single night to do that. So we're talking about sessions and not necessarily session length. And this is part of the reason that we've done things like... we want people to show what they did afterwards.

And so community engagement and feedback. We played around with comments and likes things like that. And are sort of pushing that off to the side saying, "We'd rather just have you show what you did" versus, you know, just say, "Hey, I like this idea."

And so, that's really what we're after. And frankly, it's early days! We're experimenting like crazy. We have a CEO and a head of product that are super passionate about this space. And so it's, again, it's part of our mission.

Jerry Li: This is a product that inspires life!

Jeremy King: You know, I had a lot of choices where I could have gone to work. After spending some time with Ben and a few members of the team, it's like pretty hard not to love this product. Right?

And I actually went around... I tell people this all the time, I went around and asked people... because I wasn't really a heavy Pinterest user....

But I go around and ask people like, “Hey, do you use Pinterest?”

And people will go, “oh my God, I love Pinterest! Here's my boards. And this is what I'm working on.”

I'm like, oh my God, so many people! And it's funny, everyone thinks... a lot of people say that, "Pinterest is largely built for women"

But I'll tell you, every man that I know that's in construction or a DIY person, or any kind of designer or anybody's into fashion. I mean, they all have really elaborate Pinterest boards.

And some... I came across this metal worker. He’s a hardcore metal worker that was building like benches and stuff out of this awesome metal. He has the most amazing Pinterest boards where he gets his inspiration from!

People who run movie studios, TV... that are all getting their inspiration off of Pinterest. Which is it's wonderful to build a product that people love like that.

Jerry Li: Yeah, it's an outlet for a lot of creative people.

Jeremy King: Yeah, exactly.

Patrick Gallagher: I use it. I used it to inspire.. honestly, to inspire my first home office...

Jeremy King: Yeah, me too!

Patrick Gallagher I don't have any style. Personally, I don't have any style. I don't have any aesthetic. So I just, I needed expert opinions, essentially.

Jeremy King: Exactly. Exactly. It's a perfect spot.

How AR aligns with Pinterest’s mission & allows “Pinners” to explore & experiment with different identitiessw

Patrick Gallagher: My next question, please feel free to tell me I don't have an answer to this or cannot talk about this.

But I was just curious, cause like with the mission of people discovering and then going out and applying it. I'm like, okay, cool. Like in a way, like Pinterest is like helping people in real life, enhance their real life.

Is there like a metaverse or an AR VR sort of component on the future? Are you allowed to talk about that? If not, totally fine...

Jeremy King: We haven't, got like a complete... But as I've been working with other retail and home and fashion companies, there's a ton of experiences... just like the beauty AR try-on.

And you'd be surprised... The interesting thing I love is people will experiment with really interesting colors that I think not only are fun, but also like really let them explore.Like I might even say, if you walk in Sephora, you're not like walking up to the orange and the green make up. But when they go to Pinterest, they do the sort of classic red and the base tones. And then they go straight to “what does purple look like? And what does orange look like?”

And that's where it allows people to explore in a free way. And again, a personal way. And they're not sharing this with their friends or, the world. And so they can really become themselves.And we see that from everything. Everything from home fashion decor. I mean, not everybody doesn't want the cookie cutter, home or their apartment. They want something unique.

We have this notion of private boards too. And private boards, not everybody uses that. But you know, you see lots of people who are just experimenting with maybe a new look. All kinds of different things that, you know, they're not ready to share yet with the world. And it's a great, safe place.

Covid’s impact on retaining talentw

Jerry Li: Earlier you mentioned your journey to joining, and working for Pinterest and, going back to the topic of building inclusive products. What's the relationship between that, being able to attract and retain talent and how that comes back to impact the product itself in terms of making more inclusive?

Jeremy King: Yeah, I think it's fair. It's a fair question.

One of the worries I've had about, COVID and not necessarily with Pinterest. But I'm a member of the CTO forum, which is about 150 CTOs that get together every quarter and talk about what's going on in the world.

And not everybody has a mission like ours. I just worry that the world is going to become a bit more mercenary if you know what I mean, "Hey, I'll go work for this company and, Hey, that's great. But you know, this company over here is going to pay me $2,500 more or whatever. So I'll just go over there...." right?

And so I'm worried that companies that don't have a mission like ours, are going to have this sort of mercenary effect. And it's going to change the way we think about engineering and onboarding and all sorts of things. As we talked about in productivity.

With Pinterest, I think we do attract a very, a very diverse set of engineers. But also people who want to work on these types of problems.

I know, uh, there's been a lot of PR about folks that are experimenting with this around the internet. But I mean we're putting this into action. You can see it on our website, you can see it with our features. We've got, you know, teams that are focused on it. And I just spoke at, /dev/color this weekend and it was a fascinating discussion about how challenging some of these problems are from a technical standpoint.

And frankly, engineers want to work on super hard problems. And things that they can sink their teeth in, but also can make a difference. So I think that attracts people to Pinterest.

Jerry Li: Yeah. I remember the founder of /dev/color was from Pinterest...

Jeremy King: Yes!

Jerry Li: ...working at Pinterest when the organization was founded.

Jeremy King: That's right. Makinde was the Pinterest engineer and who founded it! Yeah, it was great. Yeah.

As a matter of fact, I gave him a shout out this last weekend. It's a great organization to to be affiliated with. They do a great conference. It was great.

Jerry Li: Yeah, that's a good reflection of the culture.

Jeremy King: Yeah, exactly true.

How “investing in rest” & cultivating work-life balance can increase productivity

Patrick Gallagher: One thing I wanted to ask you. I saw last week that there, I think I saw this as a post on LinkedIn about invest in rest.

And because I know one of the things we wanted to talk about a little bit about productivity and some of the challenges and dilemmas related to productivity now in a year and a half sort of post COVID world.

But I wanted to start with invest in rest because I'm tired. I'm burnt out. I'm Jerry and I talk about this. He's burnt out. We're both like burning oil and working a lot of different things.

Can you tell us a little about the invest in rest movement? what inspired that and what's the intention behind it and how has it been going?

Jeremy King: Yeah, it's an interesting point. And frankly, it's so tough because it's hard to detect in a COVID world, too. Right? Where you, you got this great employee base that you're not necessarily connecting on a personal level on a, day-to-day basis anymore. So it's hard to detect this somebody's under stress and that's, as much as you were able to in a previous world.

And so we're not a retailer, but you know, we're a heavy Q4 kind of company. And so lots of our features are getting launched right now or doing our big launch for creators tomorrow. So our teams have been burning pretty hard.

But, as we looked at the data. And COVID, I think almost every single engineering company reported an early, increase in productivity.

And as I've done, research, not only here, but also with a number of third parties that have been investigating those by looking at people's calendars. And when they're checking in code and these sorts. You're seeing, people's workdays grow right across the board.

And it used to be okay. You work like, as an engineer, you're working most of the time you're working like an eight to nine to 10 hour a day. And now people are working like these 12 hour blocks and they might have some breaks in between, but it's a really long time to be building products and being focused and just not being away.

What we've decided to do is we've been calling them Pintention Days, intentions, like, “Hey, leave the company” you know, it's like literally extra holidays. And it's frankly, it's, made a huge difference in not only in the morale, but the productivity of our teams.

If I go take a personal day, when I come back, I've got two days where the email I got to go deal with. If the whole company takes a day off, then you’re pretty much free. That's like an extra Saturday, if you know what I mean.

So it's, it's really helped us a ton. Not every company can afford it or we'll do that.But, if you think about taking some time for yourself and giving your life in order, work-life balance is a real thing.

And so we were just worried, given where we were and the cycle that we just needed to make sure people had some time to rest, if you will.

And not everybody does that. I mean, engineers in particular, you know, they can code, I'm using, coding all night kind of sessions. And if you don't have anything else to do and your friends aren't going out, oh, I'll just, cut a little longer. Right. And it can be pretty unhealthy.

That's what we're trying to prevent from.

Patrick Gallagher: I think it's so, so relatable. Because I think Jerry and I, in different ways in different sort of expressions of that have found ourselves in places like that... where we're not really going anywhere. So we might as well work late doing something like launching this new thing.

How to schedule a day off for your entire engineering org

Patrick Gallagher: So I guess for somebody who's going to roll this out, I'm looking for your quick advice... if you were to do like a full day off for the team, how do you set that up? Like how far in advance do you communicate that to the team? How do you set expectations around...

Do you have like quick, like structural tips for how to set up something like that?

Cause I'm also like asking in case like Jerry wants to do this before one of our big launches.

Jeremy King: Yeah. Well, it's funny you say that though because we’ve had these debates because the last potential date we announced just two weeks before we did it.

And it was... frankly, it was a disaster for some teams because if you got this launch coming out... like taking two weeks, Or two days, one or two days out of a schedule that's only two weeks long can be really painful...

So I would say, give it a month to say, “Hey, you know, next month, we're going to take an extra day.”

That sort of thing.

If you do it, the next week... You cause all kinds of problems where teams are... you know, they're not ready, they can't do it. And that sort of thing.

So people can plan around it. That's number one.

Number two, we also ask people to go out and show their teams what they did on that day. And then you have people “I went to go visit my mom!” or “we went on a hike” or “I just sat on the freaking couch and watch movies all day!”

Just so you could get people to really let go versus, quietly checking and that sort of thing.And it's, been great. and frankly, it's been super well received in New York.

So, like I said, not everybody can do it. I've had to do this for years. When I was at Walmart, I didn't have a Thanksgiving for eight years. Because, you really can't work on Thanksgiving when you're a Black Friday kind of company...

Patrick Gallagher: Oh my... I didn't even think about that!

Jeremy King: Exactly.

So you have to remember you now have to go give people an extra few days to go have a Thanksgiving for their families. Right. Especially for people who are working in the NOC or, their SRE types. It's pretty hard to give those folks have the day off when everybody else is off. Somebody's got to watch, you're running a 24 7 business like we are. Not, everybody can take the day off.

So make sure you're paying special attention to those folks who don't get the day off and give them a couple of days off later.

Patrick Gallagher: Definitely. I love that.

Upcoming Industry Challenges: building in serendipity, onboarding in a remote-first workplace & the logistics of workplace flexibility

Patrick Gallagher: So you shared a little bit about solutions for people to help overcome burnout, but are there other elements of like the post productivity dilemma that you're paying attention to or observing or some of the trends that you're seeing there?

Anything else that's stood out to you the last year and a half that's changing.

Jeremy King: Yeah, it's funny. We went back to the... When San Francisco opened briefly, we went back to the office. I was going probably two and three days a week, and you know, we have about a thousand seats in San Francisco and we were only getting a couple of hundred people coming, but it's funny.

I had a couple of sessions I had a five minute meeting with my finance partner and I haven't talked to Molly in awhile. And I had this question about how this one report was done and we literally had this five minute meeting. And I'd be like, “oh, I got it for perfect. Awesome.”

And I just forgot about the five minute meeting! Those meetings don't exist anymore.

Like, oh, I have to schedule it. And nobody ever schedules five minutes and it was too long for someone. And, you know, too much pain in the ass for half an hour meeting or getting 15 minutes on your calendar, then it's getting lost.

And I'm a little bit worried about those type of moments, especially for onboarding. So we do a lot of interns, a lot of apprentices, a lot of college hires. And frankly, we know that onboarding has been more painful, less productive, than when we're in person.

So we've invested heavily in onboarding tools and lots of documentation. And lots of meetings. But it's still, even though we've invested a time, it's still not as good as like clicking over to your neighbor. “Hey, what was that... What was that system called again? And how does that work?"

Like taking that two minutes to help somebody get up to speed.

And so I'm worried about, you know, long-term productivity of new hires in particular. and whether they come up to speed as fast.

I think most other CTOs are also worried about invention and innovation. So how do you build a serendipity that an office environment has? And especially when you're, you know, we're not that big of a company, but you know, we're a thousand engineers, about 3000 employees. It's hard to like connect people.

And so we've been having sessions, we've been calling them coffee chats. And frankly, people just want to sit in a room with people that aren't on their team and just talk about the company or what's going on. Or just like connect people that literally don't work together every day in the same company, but don't work together

And they're just like fascinating. "Oh, here's what's going on with sales and, oh my God, this is what's happening over on this part of the engineering team."

It's just those kinds of sessions you need to like now physically make happen. Versus the past, that would just happen in the lunchroom or while you were walking around the hallway, right?

So I'm worried about it. I don't think there's a perfect answer for this, but you know, we've gotta keep working on it.

Jerry Li: Yeah, I can't help asking how do you make those sessions happen? Even manually?

Jeremy King: So, frankly, who figured this out was our CEO. And so your CEO sends you a note saying, “Hey, would you like to do a 30 minute coffee chat with me?”

And for other people, it's pretty hard. People don't turn that down, right!?

So he came up with that insight of initially saying, oh my God, it was why didn't get a lot of insight. But people just loved hearing from the other teams. I never get to interact with another team. And so we started that with a lot of executive groups.

And then each we have... We have a bunch of internal resource groups for our different communities. And they've started it with themselves saying, "Hey, if you will, you know, 10 of us are gonna get together on Thursday and sign up here"

And that sort of thing.

So again, it's mostly volunteer, so we have to figure out a little bit how to ingest this and into the rest of the company.

It's a, yeah, in a post... I mean, there are a lot of companies that are fully remote. you know, people love that environment. Personally, I don't love a fully remote environment.

It's hard. I even like when I'm speaking to a large group, I often will speak to a hundred or maybe even a thousand people. And you can feel the energy in the room. You can understand whether they're, getting you. You can, you know, see if people have questions and that sort of thing.

And that's pretty hard to deal with 12 square is out of a thousand are showing up on your screen, You can't really read the room. And so it's harder as a leader too. There's lots of work that I need to do. And the rest of the world needs to figure out..

Patrick Gallagher: One of the other things that you had mentioned just briefly was about like the, some of the challenges around people interpreting flexibility differently.

I was wondering if you could, could introduce that a little bit and like that. So how the challenge of that dynamic, because I think when you brought that up, I was like, yeah, of course people have different interpretations of flexibility, but that makes it really, really complicated!

Jeremy King: Yeah, this is exactly the challenge I'm talking about is... you may have one person who thinks flexibility is "I'm not coming in before noon."

And other person thinks that "I'm only working on Wednesdays and Fridays."

And other person thinks that "I'm working from Hawaii." Another person says "I'm going to work from India," And so at that point you have a team that can never connect with each other, right? This is where I think companies are going to these norms and sometimes they call it core hours and a few other things, and they're having to be pretty strict in some cases saying, Hey, we're going to work.

This team is going to work from 10:00 AM Pacific to 4:00 PM Pacific. you need to be online so that we can have some interaction. and maybe it's Monday through Wednesday, you do that. And then on Fridays and Thursdays it's open. So I think that's, what's going to happen.

I think Amazon and a few other companies have decided to make it team by team that get to decide.I think that's going to be short-lived as well, because it's going to be chaos when, “Hey, this team only works on Fridays and this team only works at Mondays.”

How are you going to get those teams connected? But, we'll see, we'll see how it goes. But I think these norms are going to be really hard to figure out.

I always make fun of folks worth wanting to work for them. Why you got to a person who works at New York, you got a person is in Hawaii. Now that you only have like a two hour window where you guys can talk to each other.

And if you're working from Hawaii, you're probably working on west coast time. Anyway, you know, you'd get up late and, you know, work later.

But we'll, we'll see. We'll see. I think all of us is going to shake out in the next 6 months

Patrick Gallagher: Yeah, Jerry. And I definitely feel that tension. We were west coast, east coast split, but today, even then like east coast, west coast, like there still is a little bit of that of like you wake up on the west coast and the east coast has already been living for a couple hours. And so it

Jeremy King: Exactly. And that happens a lot. Yeah. I'm going to spend some time in our European offices and you ended up doing double days cause you do a whole day in Europe. Then California wakes up and then you got to do a whole day with them. Right. So you're very tired when you get back.

Patrick Gallagher: Jerry's familiar with that

Jeremy King: Exactly. Exactly.

Rapid Fire Questions

Patrick Gallagher: Jeremy, we're getting close towards the end of our time. This has been a fantastic conversation, but we've got a couple quick, rapid fire questions to get into. if you're ready to dive into those.

Jeremy King: Yeah. Bring it.

Patrick Gallagher: All right. I love it.

So, first one, what are you reading or listening to? Right.

Jeremy King: Yeah. Well, I talk about hell Mary. Uh, Andy Weir at, I believe is the, that is a wonderful. I just finished. I was just on a holiday with my wife and I read, Stephen King's novel, it's a time-travel novel. I think it's called 11 22 65 or 63 or something like that?

It's all about time travel. It's great. Stephen King.

And then the non-fiction stuff I'm reading is a book called signal to noise, which is about statistics and analytics and how to, how to determine what what's lying to you, you know what data's lying to you.

Patrick Gallagher: I really admire the blend of fiction and non-fiction...

Jeremy King: Yeah,

Patrick Gallagher: a great left right. Left right. Brain balance.

Jeremy King: exactly.

Patrick Gallagher: Next question. is what's a tool or methodology that's had a big impact.

Jeremy King: Interesting. the one I tell CTOs the most is, I call it the transparency methodology, if you were...

I have the number of times in my career where I, whether I had a hundred engineers or a thousand engineers, where either my boss or the head of product comes to me and says, "you got to have 10 people that you can shake out of the street and go work on XYZ project."

And the only way to answer that is to say, "Here are the 20 things we're working on, which one do you want to do?"

Or, you know, that sort of thing.

So being extremely transparent with not only the product roadmap, what your engineers are working on, how much things can. All of the above.

I'm a ultra transparency, person, and that's just helped make decisions faster. Make sure that people know why I'm making decisions in certain ways. And it's a tough thing to do because it's a fluid environment and you have to put it in processes in order to get.

Patrick Gallagher: ... I have to ask a follow-up question here for like a scale of 20 different things... How do you typically track that? is there a tool that you use or is it...

Jeremy King: Pinterest, we use JIRA and we're migrating. It's not as easy as you think we have a process that extracts data out of Workday that shows where people are working. And then we tie that to an OKR and it's a little bit more manual.

Believe it or not. I have 10,000 engineers working for me at Walmart, and I was able to, with a click of a button, I could tell exactly what Jerry was working on. Jerry you're working on this right? And it was all. It wasn't any time-tracking. It was just, when you checked in code to get hub, you had to assign it to an OKR. And as a result, I knew what everyone was working on. it's an awesome process. There are a lot of tools that are coming out better like that, especially in the COVID world. Cause it's hard to keep track if you don't have a process like code word. So once you get over like 20 engineers, it gets pretty hard to keep track in one person's brain. Right. So, it's a good, it's a good thing to, make sure you put in your.

Patrick Gallagher: Thank you for answering that. That was because I think you've nailed it. Like tons of people are dealing with that challenge right now. S

Jeremy King: Yeah.

What is everyone working on?! That is such a common question...

Patrick Gallagher: Yup, okay.

What is a trend that you're observing or the you're following that's really interesting or something that hasn't quite hit the mainstream.

Jeremy King: Oh, interesting. I love all, forms of alternate power. I've been looking at this new battery tech. I went to a TED conference like 10 years ago and they were talking about battery tech that, you know, you could use dirt just to build these massive battery.

The other thing that's coming up lately is nuclear, micro nuclear power plants.

That's... I think that's very close. And some of that is being fueled by the machine learning engines that allow us to model out these things at a much bigger scale than we used to be able to.

So it's being powered a lot of this being powered giant machine learning engines that are allowing you to, you know, to take on problems that we knew we could probably solve, but haven't been able to touch. But can you imagine we have a little micro nuclear power plants all over? It would, be a huge win for the environment.

Patrick Gallagher: What is your favorite or most powerful question to ask others or to be asked?

Jeremy King: Oh, boy, I like telling people my story of how I got to where I am.

I do a lot of guest teaching at my Alma mater. I just did one at Berkeley.

And everybody asks me like, “how did you get your job?”

And I love telling the story about how hard work will prevail, like work ethic and hard work. A lot of people say hard work beats intelligence. But I've met some really, really intelligent people, like way smart so I won't say that. But hard work can pay off. And it's really about me volunteering and taking on new hard problems that got me to where I was. And often that was at teh detriment of family life. But that’s what got me to where I am.

Patrick Gallagher: Last question, is there a quote or mantra that’s resonating with you right now?

Jeremy King: I always tell people work hard play hard. Life’s too short of a life to, you know, get so serious about everything that you're doing. You know, decide that you're going to have all the fun when you're retired. I spent a lot of time at work and I worked pretty hard, but, when I'm with my friends and family, we turned it on, right. We were trying to have the best time as we can. We can. And, uh, it's not always hard to, to live up to that, but I think it's something I strive to do. To play really hard.

Patrick Gallagher: Well, I just want to go back to the point we were talking about at the beginning of changing a meeting locations. Cause I, I did want to point out I observed the work from barbecue, posts that you did.

Jeremy King: Yes.

Patrick Gallagher: yes, you gotta bring the fun.

Jeremy King: I that barbecue. I that's, uh, that's one of my favorite Pinterest projects. I built that barbecue. It was, it's an awesome product. I need to do an idea pin on that one. Cause it's a, it's pretty amazing project.

So I'm very proud of that space.

Patrick Gallagher: Perfect, Thank you. This has been an incredible conversation. Thank you so much for your time,

Jeremy King: So glad to be here and thank you guys for making the time and take a look at our launch tomorrow on 10 20, by the time this gets published, it'll be well out. so yeah, thanks for letting me talk a little bit about Pinterest and what's going on in the world.

Patrick Gallagher: We covered a lot of different topics with Jeremy and this episode, we wanted to focus these takeaways on where you should start to build more inclusive products. So here we go

First check to see if your data is biased and most likely already is. If it is biased, you'll need to invest in filtering your data set or buying and acquiring new, more diverse datasets.

Listen carefully to your user base, especially to new people starting to use your product and dedicate resources to experiment, test, and solve the problem.

One final consideration. We've brought up burnout a couple different times in the last few episodes...

For a lot of folks, the holidays and the new year invite all sorts of deadlines and pressure. I know I felt the same way. Make sure that you're taking care of your teams and don't be afraid to invest in a little rest.