SPEAKERS

Lusen Mendel, Director of Developer Relations @ Karat

Lusen Mendel (they / them) is Director of Developer Relations at Karat, where they share interviewer best practices with interviewers around the world. Previously, Lusen helped build Karat's Interview Engineer organization. Before that, they managed engineering teams at Indiegogo and Rackspace, worked as a software engineer at a number of startups and research groups, and graduated with a couple of Computer Science degrees from MIT.

Lusen is also the host of Candidate Planet - a podcast / YouTube channel empowering engineering candidates to ace interviews, negotiate offers and advocate for themselves at work.

"Candidates are so vulnerable at the beginning and end of the interview. And the beginning is when they're getting all this new context and they're not into the swing of it. And THAT'S the time to make sure you're removing noise. Because if the candidate is answering the wrong question... it really just is a waste of a very limited amount of time."

<cite>- Lusen Mendel </cite>

Rohit Grover, Senior Interview Engineer & Mentor @ Karat

This Episode Is Brought To You By

Jellyfish

Jellyfish helps you align engineering work with business priorities and enables you to make better strategic decisions. Learn more at Jellyfish.co/elc

Bugsnag

Bugsnag monitors application stability so you can make data-driven decisions on whether you should be building new features, or fixing bugs. Check out their 2nd Annual Application Stability Index report at Bugsnag.com

Special thanks to our exclusive accessibility partner Mesmer!

Mesmer's AI-bots automate mobile app accessibility testing to ensure your app is always accessible to everybody.

To jump start your accessibility and inclusion initiative, visit mesmerhq.com/ELC

Show Notes

• Setting the Stage - Interview Example #1 (1:41)
• Time management during the technical interview (5:52)
• Your 3 main responsibilities in the technical Interview (6:29)
• Setting candidates at ease when they're having difficulty during the technical interview (8:26)
• Interview Example #2 (8:58)
• How to get the best from candidates during the technical interview (11:17)
• Sources of noise & bad signal in the technical interview (16:48)
• How to give hints appropriately in the technical interview (19:15)
• Interview Example #3 (21:04)
• Creating consistency in your technical interview experience (22:12)
• How to handle a difficult technical interview (25:07)

Transcript

Setting the Stage - Interview Example #1

Patrick Gallagher: Lusen, Rohit welcome. Thank you both so much for joining us.

Lusen Mendel: Hey, thanks for having us. We're really excited to be here.

Rohit Grover: Lovely to be here.

Lusen Mendel: And Rohit, you might want to give a little more context about what you do with Karat and, you know, as an, as a senior engineer at other company, at another company.

Rohit Grover: Well, I interview candidates and I love this participation that I get from them. And I'm simply a facilitator and I let them be and show themselves in the best light. And yeah, I just enable that.

Lusen Mendel: Yup. Awesome. And and you're also a software engineer.

Rohit Grover: Yes.

Lusen Mendel: Okay, so hi everyone. Here's what we'd like to do. I'm also a software engineer, but now I do a lot of talks and workshops on interviewing best practices. I haven't written code in many, many years. Well, that's not quite true. We were always sort of writing code in the background, aren't we.

But here's what we'd like to do Rohit and I, we're going to demonstrate some interview interactions. And what we'd like all of you to do is put on your observation hats and let us know what we're doing wrong.

I'm going to do the first I'll be the interviewer for our first little role-play cause we don't have time to un-train all of Rohit's niceness. So we'll start with an easy one. So let's do that. So, Rohit will be the CA the candidate, I'll role play the interviewer. And action...

Okay. Yeah, like I said, we'll start with an easy problem. What I'd like you to do is to just write some code that takes the list of numbers as input and returns it without duplicates.

Rohit Grover: Okay. Hmm. Can I talk through my thoughts?

Lusen Mendel: Yeah. Sure.

Rohit Grover: Okay. What I'd like to do is go through all the elements maybe a loop, a four loop nested loop. Hm.

Lusen Mendel: Oh, wait, hold on a second. Nested four loop. What's the time complexity for that?

Rohit Grover: I'm sorry I'm a bit rusty about time complexities.

Lusen Mendel: I thought you had a computer science degree? You know what it's okay. Just go do what you were, you were planning to do.

Rohit Grover: Is there a better way to do it?

Lusen Mendel: No, it's it's fine. Go ahead. With, with what you were planning.

Rohit Grover: Okay...

Lusen Mendel: Okay, so we're gonna, we're gonna freeze frame and Rohit's going to implement a solution with nested four loops. We're not gonna actually code that up right now. And we're gonna pick back up as Rohit finishes typing the last character for his code to work.

Rohit Grover: Okay. Okay. It looks like it's working.

Lusen Mendel: That looks like it returning a list of the duplicates. And that's not what I asked for.

Rohit Grover: Oh, maybe I misunderstood...

Lusen Mendel: yeah, I, I wanted a function that took a list and removed the duplicates and returned all the unique elements. You're returning the duplicates.

Rohit Grover: Okay. Can I modify this?

Lusen Mendel: You know, I'm sorry. We're, we're actually at a time.

And cut. How'd I do as the interviewer, what did I, what did I do wrong? And you know, I'm hamming it up for the applause. But it is very easy for interviewers who are nervous or inexperienced, or maybe distracted by other things that are going on, you know, just coming to this meeting and their head's not in the game. You know, it's easy for us to unintentionally do things without realizing sometimes. If we're not really calling out what we should be doing. So what, what should I have done instead? Or what did I do wrong?

Rohit was this experience like for you as a candidate?

Rohit Grover: Well, I felt I felt challenged in a way that was too personal and I felt that I had to do something which I couldn't. And instead of focusing on what I could do I was being... i, I felt like I you wanted something that I couldn't give you. And so I was discouraged from doing what I wanted to do.

Lusen Mendel: Yeah! And also it's how it's particularly dismissive and probably alarming to the candidate when you say, well, you know, you're trying to engage with how to do something better and I'm just like cutting it off. Like, nah, just do whatever you said before.

I mean, what a missed opportunity as an interviewer to be like, well maybe if I explain this a little bit, what complexity is and then see the candidate apply that or have some good insights. Like that's actually a nice opportunity that I could have taken advantage of. And instead now he just feels like he's failing and it's that was, yeah, let's see distracted at the start. Yes. Pretty dismissive up front, but you had a CS degree.

Time management during the technical interview

Lusen Mendel: Ouch. Let's see that felt like a micro aggression guide them up front and help them to make the most of the time.

Yeah! I mean, this is the worst part of the interview. Is that we spend all this time and then at the end, Rohit gives the right answer to the wrong question, which we waste both of our time!

Like I have one job as the interviewer. My one job at the end of this very limited time is to have a yes, no decision. Do I think Rohit should move forward or the candidate should move forward?

And like, I didn't even accomplish my own goal. So time management is really, really important.

Rohit, do you want to say more about this? Cause I know, I know there's some things at Karat that y'all do to really manage the time well.

Your 3 main responsibilities in the technical interview

Rohit Grover: Yeah. So, at Karat well, when I interview, I, I try to make the problem statement as clear as possible as early as possible. So, I let the candidate show what they can and it's, it's more important as to what they can achieve. And in many engineering scenarios, it's okay to get an acceptable solution and then improve upon it iteratively.

So that's completely fine. One, doesn't have to point out that something better can be done right at the beginning. So yeah. Let the other person be and show what they are able to do.

Lusen Mendel: Yeah.

And in particular, so one of the things we talk about at Karat is that every everything you say and do as, as an interviewer, maybe this is a simplification, but I like to say it falls into three categories. You're either clarifying the question or what the candidate is supposed to do. Like as interviewers, where guides we're hosting this whole conversation. So,we're either clarifying it, or we're providing encouragement, like being a nice host. Or we're giving guidance or some kind of hint that's specific to skill assessments you usually right.

And I like to break up, you know, it's important to pay attention to each of those three different things like encouragement is free in a sense that's, it's just like being, just helping someone feel at ease. Versus guidance might be when we get into that situation where, okay the signal has, has dropped. The candidate gets stuck.

And there's lots of different things we can do. You don't have to give guidance, you can ask a different question. You can remove a constraint that, you know, there's different things, but if you are going to start giving help, sometimes that impacts the assessment. Right?

So separating out saying something that, you know, just because you're not giving guidance doesn't mean you have to be silent. But that clarification step, that first, one of being clear to the candidate, I think that's the most crucial upfront. That candidates are so vulnerable at the beginning and end of the interview. And the beginning is when they're getting all this new context and they're not into the swing of it. And that's the time to make sure you're removing noise because if the candidate is answering the wrong question, it really just is a waste of a very limited amount of time.

Setting candidates at ease when they're having difficulty during the technical interview

Lusen Mendel: Okay, should we try? Is there more we want to say about this? Let's see. Yes. Cynthia, I liked that you said that when the candidate doesn't feel at ease, it's hard to assess them properly.

Rohit is there anything you do in particular to set candidates at ease? When you can tell they're having a hard time?

Rohit Grover: I acknowledge what they have achieved already. And it's more important that they feel comfortable about themselves about their person. So yeah I just turn towards them and acknowledge what they have offered and yeah. Things, things just fall into place.

Interview Example #2

Lusen Mendel: Okay. Cool. Cool. Cool. Well, let's, let's try this again. Why don't you be the interviewer? Now that we've gone over some of these good tips, I'll be the candidate. And action.

Rohit Grover: Hi, how are you doing? It's really lovely to meet you.

Lusen Mendel: It's nice to meet too.

Rohit Grover: So, so today you're going to do a programming task. And what I'd like you to do is to write a function. And we'd like this function to take a list of values. so let's say we've got a list, let's say 2, 3, 2 in that sequence. And this function needs to take this input and remove all the duplicates.

So in this case, with this particular input, the function would return two and three, because those are the unique elements after removing all the duplicates.

Is that clear?

Lusen Mendel: Yeah. That makes sense.

Rohit Grover: And so what I'd like you to do is to take some time to think of an approach. And then once you have an idea, please discuss it with me.

Lusen Mendel: Yeah. Okay. Yeah, no, I'd like to think about this for a little bit. Is that okay?

Rohit Grover: Sure, sure.

Lusen Mendel: All right. So I think what I'll do is I'll loop through the input. And I'll create a hash map. So as I loop through the input, I'll put the values as keys in the hash map. And then values I'll count how many times I've seen that key that input before.

Rohit Grover: Uh Huh.

Lusen Mendel: So I'm not really sure that w how I can use that though, to then return a new list... I think I'm missing something.

Rohit Grover: So remember that you need to start with the list. And returning another list which has the same elements but only the unique elements, which means the duplicates need to be removed.

Lusen Mendel: Yeah, no, I got that. Yeah, I'm just not certain how I can... it's been a while since I've used hash maps, I'm just not certain how I can use this, you know, to do the next step. I just can't figure out what that next step would be in this approach.

Rohit Grover: I see. Well, I can't really tell you that. We've got so many candidates and I don't want to put you at an advantage by telling you what the next step would be.

Lusen Mendel: Oh, okay. Yeah. No, I understand. That makes sense. I'm just, I'm not, I'm not sure what to do next.

Rohit Grover: All right. Hmm. So you've talked about building this hash map. And you'll have counts associated with each value. So, what we want you to do is to return a list. That's similar to the original, but with the duplicates removed.

Lusen Mendel: Yeah.

I got that. Okay. I want to cut before I get too exasperated.

How to get the best from candidates during the technical interview

Lusen Mendel: You know, so this is an interesting situation, right? Like Rohit was being very nice. That was a very friendly interaction. Sometimes candidates get stuck.

What could Rohit, or what could an interviewer do in this situation? Let me just say on this, one thing that was really nice was that Rohit gave an example of the question. That's this part of being clear and really trying to reduce that noise of misunderstanding.

And sometimes if it is, you know, these are toy questions a little bit... if it is a more complicated situation, actually having the interviewer making sure to ask the candidate. Okay here's some input like, so the interviewer explains one example case: this input, this output, and then ask the candidate, Hey, you, you told me why does this input get to the second output?

Just to verify, cause sometimes candidates, You know, they're, they're nervous. They're just saying yes there, you're just trying to engage them. And that, that verifies for both of you that you're on the same page.

And I know when I do workshops with, with hiring managers and interviewers and I always ask, you know, what are you looking for in your candidates? What are some of the competencies and skills? You know, how do you assess them?

And pretty much always there's at least one person, sometimes, sometimes everyone who's saying, well, I want my candidates to be able to handle ambiguity. Or I want them to ask me questions or explain their thoughts unprompted. Or, you know, ask questions unprompted or, you know, write tests. You know, there's sort of almost like looking for the things that the candidates do themselves. As if, if they tell a candidate to do something that's cheating.

And I want to draw very important distinction here, which is that the absence of behavior does not let you assess that. So if you want to assess someone's test writing and they don't write tests, it doesn't mean they can't write tests. It just means in this interaction, they didn't think that was the right thing to do.

And candidates get all sorts of weird advice. You know, there's, there's interviews that are about getting to the right answer as quickly as possible. There's interviews where it's about the conversation or the code design. And so ideally you're telling your candidates up front really what the objectives are because you don't want your candidate to make a decision that biases, you know, that, creates noise.

Because candidates, you want to be the one making the decision, right? you want to control that.

So don't let your candidates, don't put your candidates in positions where they are unintentionally making noisy decisions. So telling objectives is one thing, but even, even, I'm just going to give this tip right out here... to specifically ask candidates what you want them to do.

If they've written some code, if testing is on your rubric or part of how you're going to evaluate that yes, no decision. Tell the candidate. Okay. Now I'd like you to come up with a test suite, just tell me, you know, what are, what are some of the things you would, you would test to build confidence in this code or that you would put into a refactoring? Or whatever.

Then you can get the signal of how well they do. How well did they write tests? If you want to evaluate ambiguity, how well someone like handles ambiguity, tell the candidate you know, something upfront a little bit like this is, I'm going to give you a vague question and we're going to, you're going to need to ask questions and we're going to need to have a conversation to figure out exactly what you're doing.

And then see how they handle that.

Sources of noise & bad signal in the technical interview

Lusen Mendel: Rohit, is that, what do you, what do you think?

Rohit Grover: Yeah. So important to identify what they are being asked. And as you said, that that's the signal. So I clarify as many times as needed, what is being asked. And in a way that's friendly, that appears to be confirming.

And it's like an affirmation. "Yes. This is what we need to do. Yes. I understand. It is, this is what we need to do." and then they can demonstrate what they're, they're able to perform in that situation.

And yeah, just, just being that presence of support, protection, and just presence with the right amount of information at the right time. That is what helps me get more out of my candidates.

Lusen Mendel: Yeah. Yeah. So there's, I would say two big pillars of introducing noise in a negative way or, or, you know, not seeing the signal. And so one of them is basically candidates answering the wrong question or getting off track and and so. Thinking really thinking a lot about that, that how you ask questions and, and how you clarify is really important.

There's another thing that happened in this interview, which is that the candidate got stuck and we wasted time. Like I was just, you know, you know, sort of the toy situation, but you know, it happens right that candidate, they don't know something, they get stuck.

And like I said earlier, there's different ways you can handle that. Sometimes it's like, Hm there, you know, this candidate is having trouble with this data structure. Let's move, let's try a different, let's just try different problem.

And, and part of this is like, I used to do a lot of engineering interviews, you know, I can, I could ace five interviews in a row for, you know, for coding interviews and then like bomb on the sixth one.

And is, is my ability as an engineer, the average of all of those? Or is it, you know, it's usually not. Yeah. Right. Like the competencies don't really often work that way.

So you're trying to, typically, you're trying to find the candidates best performance. Especially for the skill assessment. Skill assessments or a reaction to the more conversational interviews where you could like confidently explain someone else's architecture or explaining some other work or future work. You can confident your way into a false positive, a positive decision to move forward from the interviewer that maybe where you're not actually a good fit.

Skill assessments, you can't usually confident your way into like knowing an algorithm you don't know. Confidence still helps always. But... but you can have this, like the false negatives become the danger of bad performance and getting nervous or, or even just simply like bad time management.

How to give hints appropriately in the technical interview

Lusen Mendel: So with the hints. Yeah. So in this case, in this demo, there are things like Rohit what would you have done if this was a real interview? This was a real question I was getting stuck. What are the guideposts for knowing, you know, what, like how to go about giving hints?

For some, for like an organization that doesn't have anything written down to interviewers and interviewers are just like each doing their own thing.

Rohit Grover: So, so in this case, your solution did have some merit and, and if I could see a way to help you complete your solution, I would have nudged you gently towards it without actually giving you the answer.

And otherwise I would reiterate the part of the question that really mattered, which may have been the point where you strayed away. And, and bring the focus back on to that point where at least we had some alignment between the approach and the problem.

So either reinforce the point of departure or encourage the part in the candidate's solution that could still lead to a full answer. So, at the same time, I didn't want to hurt their dignity by giving them the answer because well maybe yeah, that's, that's not fair to them or to the signal. But yeah, just coming back to the, the most useful point that we had achieved without actually calling it a hint. Just, just reinforcing that.

Lusen Mendel: The devil's always in the details here, you know? Like how do you know when someone is stuck? There's different ways to do harm. You can do harm through inaction or over action. Right?

I like a lot of these, I'm always wearing my engineering hat. It's like, always about balance and trade-offs right. And always trying to find the sweet spot with, with all of these points, encouragement, clarity, giving guidance. You're always trying to figure out like, kind of read the room with the candidate.

So maybe we'll do one more demo. And then that, I think that's an interesting question to answer. You know, some of the more tricky cases. And I have one, I may have one tip. I want to make sure I get in before this is over.

Interview Example #3

Lusen Mendel: Okay. Let's do this third one. How about, I will be the interviewer, Rohit, you'll be the candidate. Okay. Action. All right Rohit you've been doing great so far. For this last bit, what we're going to do is a coding exercise.

What I'd like you to do is to write a function that takes in a list of numbers and returns that same list, but with all the duplicates removed from the original list, does that make sense?

Rohit Grover: Yeah. Okay. Hm. Quick question. Does order matter in the return?

Lusen Mendel: Oh, order yeah. You know, you could sort it like either way is fine.

Rohit Grover: Oh. Oh, okay. That sounds good. I'll start by sorting it. Then why don't I loop through every value in the sorted array. And then for every element, I'll compare it to its adjacent value.

Lusen Mendel: Yeah. Cool. Yeah. So be doing like that sort of thing. And then we get to the end, you know, that's going to be like, you'll probably have a little edge case there. Right? Go off the plus one boundary.

Rohit Grover: Oh, oh, yes, yes. Yeah. I'll definitely handle that.

Lusen Mendel: Yep. Cool. All right, cut. An overly friendly interviewer...

Creating consistency in your technical interview experience

Lusen Mendel: I mean, this was just to sort of highlight the difficulty that can happen when the inconsistency that can happen. When you're scaling an interview program about how, you know, your interviewers are juggling, all these things are trying to connect with people. They're trying to be encouraging. They're trying to figure out what is noise? What is signal? What do I let the candidate do? When do I step in? But what, you know?

And so in this case, it kind of, right. I felt like I gave, I gave, I like mentioned sorting. I mentioned some edge cases. And at the end of the day, there's no right or wrong answer, like different companies and roles might require different things. They should be probably, I would just say, write things down. Like write down on your question guides, what is considered the, you know, the signal part of the question and if there's any noise.

Like maybe if some questions require string manipulation, but that's just too many, you know, because of handling input and that's not as important as some of the like algorithmic components or convert, you know, whatever the competencies are, maybe on a rubric that they're also written down.

Tell the interviewer, if the candidate doesn't know this, like give them the answer, tell them to look up docs, give them the split function. Like don't spend 10 minutes on the unimportant thing.

So tell them what is noise and the same, telling interviewers so they know what noise and signal is. It gets really important with the hints so that they know. You know, if you don't have any documentation for interviewers, just, you know, write some bullet points and have something general about like inner technical interviews in general.

But eventually it can be really helpful on specific questions to say, you know, here's the common mistakes and here's how you could ramp, you know, for each, maybe there's different approaches. Here's how you can ramp them up in different ways.

That it just enables interviewers to succeed when probably your interviewers are actually engineers. Like this isn't even their jobs. Like they're coming to the meeting, like you're coming to the interview. And if they have a guide that tells them there's different approaches, then they're just going to be so much better at interacting with candidates. I mean, hopefully they've read that guide previously, but even in that moment, they're not going to like shuffle candidates, you know, to one solution.

So they'll know different ones. They'll know here's the progression of questions. And even here's how that impacts the assessment maybe.

So writing things down can be really, I think, really powerful for consistency. That's I mean Rohit, you'll probably agree with me on this. Hand-holding is one of the guidance hints is one of the most difficult things to get consistent.

Like even at Karat we spend a ton of time doing, even practice interviews to see that inner, to see that an interviewer against, you know, this kind of role-play that Rohit and I are doing. We'll do that in a, in a more real like interview setting.

And you can imagine doing that where, you know, the candidate is trying to try to have certain... maybe they're they're competent, but they've never done this interview before they're making assumptions. Does the interviewer detect that as noise? Or does that influence the signal or, you know, they're, they don't know certain, certain competencies. Is that coming up? Are the rubrics looking the same? So you can, you know, you can kind of see how you if you really want to invest time, time in this, you can really have like a close the loop.

But writing things down, even a few bullet points, that's always helpful.

How to handle a difficult technical interview

Lusen Mendel: Oh, here's one! What's like one example of a difficult interview to get a good signal out of and like what, you know, what's a trick for that?

Rohit Grover: So in some cases, candidates have a particular approach that they are fixed on. So they just want to force themselves through that approach. And maybe they've seen a similar problem. They have seen a similar solution and they just want to apply that at all costs.

And so it's difficult to make someone realize that they have perhaps too much fixation on a particular thing without giving giving up too much of the information. So, stay with them and remind them what the main problem or the main task is.

But if, if they have not been able to understand that all I can do is to reiterate the task. And the next step for me would be to actually point out that they have a, they have an error. So the, the best I can do is ask them to explain how their solution would work with the particular example.

But if they even don't get it even then, then it becomes difficult for me to get out of the situation without giving away the whole solution. So some people are fixated.

Lusen Mendel: You know what that's interesting. you could imagine a candidate performance. We record all of our interviews for internal trainings and to it, to share with the hiring managers and recruiters that we're doing these interviews for.

And you can imagine two interviews with, you know, just without just out in the world, without any conversation between them. Look at that same performance, come to two different conclusions. Oh, this was a great conversation. And they nearly finished, like, yay, let's hire them.

And someone else is like, whoa, they didn't get it done. We're results oriented. They didn't write tests. We shouldn't hire them. Know they wasted time talking.

And who knows what the right answer is. But they're they're inconsistent. Right? Those two interviewers saw the same performance came to different conclusions. So, there's a lot of nuance here, like even in that situation, Rohit you're giving about, oh, someone's fixated, you know, is the competency, brainstorming is a competency writing code?

And trying to make even more specific questions so that your interviewers can, can really partner with the candidates so that everyone is in the clear. Like that clarity point again, and knows exactly what we're trying to do.

You could even imagine the question where you give, you give an algorithm and see them write code. You ask a bunch of questions for them to give approaches to, and then you choose one that they understand to implement code rather than sometimes we give these really big questions and we intermix all these competencies and it kind of shows up in the rubrics in very... in difficult ways.

So, there's no right or wrong answers. You're gonna have to figure out what works for you. Write it down so that everyone's on the same page.

It kind of got away from candidate experience, but I think all of this, you work backwards. You know, so that, so that then it's very clear what the interaction needs to be to, for everyone to succeed.

And, you know, you're never going to get your candidates... This is a funny customer service situation because most candidates aren't going to get the thing they want. But if you can help them feel respected and good about the experience, even when they don't move forward, I think that's what a great achievement and that's what we should be striving for.

Patrick Gallagher: That was an incredible conversation. Thank you both so much really quickly. Do you have any final, any final thoughts you want to share? Any, any closing statements you wanna make on the world of interviewing and effective communication for the community.

Rohit Grover: I'd say trust in the candidate, trust in this whole process where people can engage and come together. I would say trust.

Lusen Mendel: Yeah, that's been echoed, I think a few times in the sessions here, but that's really a big one. Thank you.

Patrick Gallagher: Wonderful. Great words to end on. Thank you both so much and we'll see you next time.