Cat Miller walks upright and has bilateral symmetry and opposable thumbs, which she uses to navigate the technology world as a VP Engineering at Flatiron Health.
Vinithra is currently the Head of Platform Engineering at Airtable. Prior to this she spent ten years at Cloudera, where she started as an engineer and grew into a Director of Engineering. Over her career, Vinithra has learnt several leadership lessons - through observation, experimentation and making mistakes. She is passionate about continually applying her leadership skills to high impact areas. Among the top of that list is recruiting and making it an efficient process that values everybody's time and brings out the best in people. Vinithra holds a Masters in Robotics from Carnegie Mellon University, and a BEng. from Cardiff University in the UK.
Keng has been at LiveRamp for five years and most recently holds the role of Senior Director of Engineering. He owns the technical vision and roadmap for LiveRamp Safe Haven, a global SaaS platform for privacy-first data collaboration. Keng established the Safe Haven engineering team and grew it from 14 engineers to 70 engineers globally within 9 months. Prior to that, Keng led engineering development efforts at fashion e-commerce companies including Gilt and BaubleBar.
Keng holds an MBA from Columbia Business School and a Masters degree in Computer Science from Columbia University. In his spare time, he likes to read books. Keng lives in New Jersey with his wife and two kids.
Jellyfish helps you align engineering work with business priorities and enables you to make better strategic decisions. Learn more at Jellyfish.co/elc
Bugsnag monitors application stability so you can make data-driven decisions on whether you should be building new features, or fixing bugs. Check out their 2nd Annual Application Stability Index report at Bugsnag.com
Mesmer's AI-bots automate mobile app accessibility testing to ensure your app is always accessible to everybody.
To jump start your accessibility and inclusion initiative, visit mesmerhq.com/ELC
Patrick Gallagher: Welcome everybody. Thanks for coming. How are you?
Keng He: Hello. Thanks for having me.
Vinithra Varadharajan: Okay.
Patrick Gallagher: Awesome. Well, we'll take it away. The floor is yours and dive on in Keng.
Keng He: Yeah. Sure. Of course.
I think when we talk about hiring and scaling and optimizing as engineers we'll first ask. Okay. What is the current scale? Right. So, Vinithra and Catherine, could you give us like some sense of like, what are the size of your engineering team at your current company right now?
Vinithra Varadharajan: Sure. Air table is currently at a hundred and twenty engineers and a significant portion about 40 to 50% of that was hired during this remote time the past year.
Cat Miller: We're about 300 engineers roughly and we did have a period of scaling where we were doubling every year for a number of years.
Keng He: All right. So for 120 engineers, I'm curious about what is the hiring process look like, what are the stages and sessions that you have?
Vinithra Varadharajan: Yeah. So we have a great recruiting team who we partner with and so we have sources help with the sourcing then we do some fish calls to understand the candidate better. And sometimes that's an engineering manager.
Next is a phone screen or a take-home exercise depending on what the candidates prefer. And then there's the onsite. And sometimes we do, especially in this time, of everyone being remote we are also able to support two day onsite. So you can split your full round. so yeah, and then it's the, it's a debrief and offer and sell calls.
Keng He: Got it. What about you, Catherine?
Cat Miller: Yeah, it's really similar. I'd say that we typically only do sales calls for sort of senior candidates, because at a high volume, it can be quite overwhelming. But very similar process, including kind of splitting up the evals potentially and doing a reverse interviews after we extended an offer.
Keng He: Got it. Got it. So generally speaking, we want to avoid pre-mature optimization. So what are some of the early signs that indicates, oh, it is time to optimize or, oh, it is time to scale the current process does not work for us.
Vinithra Varadharajan: Yeah. I can take a stab at that.
So the way I think about hiring process and like, especially when it comes to scaling hiring process is that you want to think of three things.
One is efficiency. So, are you valuing everybody's time; interviewers the recruiters, and most importantly, the candidates time.
Next is, are you able to hire for the profile that you want?
and the last thing is the candidate experience.
So when you think about all of this, so when you start to get a sense of one of these things is not fitting. then that's kind of like a signal.
But to break it down a little bit typically. You know, of course, if your high needs go up by an order of magnitude then that's obviously the first sign. Of if interview load is high and you get complaints from your interviewers, then that's another. And the most poignant is that you're unable to close wrecks in time.
While previously you were able to close your racks and and if... you didn't have to think too much of the process and it worked for you, but now suddenly your backlog is pretty high and that's your signal.
Cat Miller: I'm going to argue maybe a little bit more extreme position, which is, I think there's actually no time that premature optimization is possible here. Or to say it another way, as soon as it's possible that you will have an interview loop that is not exactly the same people that's been on every other interview loop, you should consider scaling your processes.
And I think that because you start to immediately get inconsistencies in the way that you've built out your process and I don't think you have to do all of the possible scaling things right away. But as soon as it's not just you and your founding friends doing the interviews, I think it's time to start putting process in place and building out what the scaffolding for your structure will look like.
Keng He: Right. I do think that consistency is like something very important, right? Like whilst we do have like 20 interviewers or 50 engineering interviewers in the interview process, And how do we make sure that different interviewer panels and that calibrate with each other and hold the same hiring bar for the candidate, for the candidates that go through the pipeline.
So, Catherine, I'm wondering if you have like some particular like tips and tricks or like, how do you make sure the bar is the same?
Cat Miller: Yeah. I mean, I think there's a number of tactics here. So the first is really to have structured interviews.
So the more that your interview itself is structured, the questions are standardized and your competencies are clear. Now you at least have a situation where your interviewers are at least trying to interview for the same thing.
And now you might just have a calibration issue. Part of what you can do for calibration is make sure that people are shadowing reverse shadowing on interviews before you set them loose in the wild by themselves.
And then something that really helped us, especially in the early days, was having a a hiring committee where people actually sat around a table and discussed every candidate. So there was still obviously a hiring manager making the decision. But particularly when you're thinking about being in that maybe 15 to 20 engineer category, it means that every engineer is going to hear a lot of that discussion and get kind of calibrated even just by hearing what that conversation is and the thoughts and the hiring manager expresses.
So I think those are three kind of easy techniques to start helping on this process.
Keng He: Uh, Vinithra, do you have anything to add here? What about your process there?
Vinithra Varadharajan: Yeah, a lot of similar things. I think what Cat was referring to like debriefs where the engineers are involved and then there's a separate hiring committee.
One of the things that we do at Air Table is that we do write down the arguments for why we should hire an engineer.
And also like the most important part in that is the leveling. So, assuming you do already have levels and cap, you know, internal strata within the engineering organization. Then thinking about who fits under what level is an extremely important and nuanced discussion.
And so being able to write it out, and of course, you're going to be able to standardize that as you get many more data points. But having that written down is a really useful exercise.
Keng He: Yeah, I completely agree with you. So...
I think the engineering leveling is something that could be used for many different purpose within the organization. But I'm I'm wondering if you implement some sort of a score card, like candidate scorecard rubric, and they say, oh, for this particular session, we are assessing this like five or six different traits.
And for those traits, like what does good mean? What is bad mean? Like, just curious about like, does is that do you do those rubrics and does that help?
Cat Miller: I think it's essential. I think those rubrics evolve over time. So they've gotten, definitely gotten more more rigorous over time. But I think that when I talk about competencies, that's a lot of what, I mean, like, this is what we're looking for. This is what algorithims... doing well at algorithms means or doing well at problem solving means here are the things that we're looking for here are the things we're not.
And for our technical questions, we also break down sort of how much progress is good, how, you know, what kind of hinting is acceptable, those kinds of things.
Vinithra Varadharajan: So at Air Table, I've seen an evolution of so initially the panels were all , so the focus areas was agreed on. However, we didn't have detailed rubrics until very recently. And that came about because again, our scale. And we have to really standardize it. And the debriefs were not enough of a standardization across the multiple groups that were emerging with engineering.
At Cloudera it was quite different in that, each... it was somewhat... it depended on the engineering manager or the hiring manager a lot. And so over there that standardization happened at the hiring manager level. and it became more of a, how do you standardize across that particular group?
And that had pros and cons in terms of what a candidate of one group might we had to put in some improvements in place so that a candidate of one group could also operate on another group. And that group welcomes them with confidence.
Keng He: Right. So we talked about... I think like a lot of this involved training interviewers and sometimes like training the hiring managers. Right. And especially for the interviewers, like sometimes the interviewer needs to shadow multiple sessions and then learn about the whole interview interview skill sets before they become, "Hey, I can be the interviewer for this like particular panel"
I was curious about like sometimes engineers are like "I just coming in, wanting to go in and develop product. I don't want it to the interview."
And as the company grow there are times, it's just like, we cannot find enough interviewers to do the interview. How do you encourage the interviewers? The engineers to interview?
Vinita. Maybe you can take this first.
Vinithra Varadharajan: Yeah. So. We haven't had too much of an uphill battle in my experience because engineers care about who they work with. So they do welcome participation or their own participation in it. One of the things that might be is that there's apprehension. So training goes a long way.
The other part is that you do have an offer, you can offer different modes of interviewing. So do you want to be part of the phone screen or the take-home or the onsite different, focus areas?
Typically, what I've found is engineers would prefer to pick a particular topic and stay focused on that from candidate to candidate. That also helps them calibrate for their own question and their way of asking the question.
And lastly I would say like, you know, acknowledging they're there, the time that they put in. Putting a cap on the number of interview, working out interview hours taking into consideration that they have to also spend time to write things out.
And so on.
Keng He: Okay.
Cat Miller: I'm going to actually say we've taken the very extreme approach of saying, "why is it, why would you make it optional?"
Everyone is required to do interviews. We don't require them right away. We want people to onboard successfully without worrying about that. We want them to get to know our company and be able to talk about it.
Um, So typically it's six months to a year before they join. But I actually think that it's a potential problem if you allow people to self-select because then you get the... you get the people who are most enthusiastic about the company and sort of your best cheerleaders interviewing, which, which sounds great.
But then they're also taking that time away from their day-to-day work. And maybe they're gonna suffer on the performance review side because they haven't been able to do as much engineering work.
But I think it's, I think it's very important to spread that around to everyone. And not make it a purely optional opt-in thing. Though I will second, you should absolutely have maximums. Ours used to be four a week, which was really heavy. And that's part of the reason we had to make it mandatory is we needed every hand on deck.
Keng He: Yeah, I think that's a very good point. And now I don't think that like everyone would make a very good interviewer right. When it comes right. Like sometimes like, do you when you bring people in, do you say, Hey, is this a good communicator? Does the candidate actually have like interview or like trick. Right, because for some of the candidates that well, for some of the interviewers, they be, they may be a little bit shy. They may be a little bit of like, you know, like not paying too much attention on the interview part of things.
Cat Miller: Well interviewing isn't an inborn trait. Any more than being a good engineer is an inborn trait. It can be learned.
And so I, I reject the belief that there, that there are people in capable of becoming good interviewers. There are people who stalwartly refuse to become good interviewers. I will acknowledge that. And sometimes you have to figure a solution for that.
I've absolutely had to tell people. Your job is to be nice. Like you need to make people feel comfortable and that is your job. You are not their adversary. You're actually, they're their biggest proponent in this moment. And you're trying to give them the best possibility to succeed.
So I've literally had to tell engineers like very basic things about like how to be friendly to others. But again, I think that is actually a really valuable skill for them to learn and to evolve. And I do not believe that you that you shouldn't... for us anyway, we're not going to hire people who don't have the ability to at least learn that skill over time.
Vinithra Varadharajan: I think part a lot of it is interviewing as part of the engineering culture. So, who you hire is also who you make your future interviewers.
So, I'd say that if you're watching out for people who do care about the larger company who do care and who, you know, have put in an effort to improve their communication skills and build that over time. Then that's going to also reflect in interviewing To Cat's point, it is important that engineering managers and hiring managers consistently emphasize that what we're looking for is we want to find what the candidates strengths are, rather than seek out all their negatives.
If you are on a hunt for that, then that in itself is going to be... the candidate is going to feel judged and nobody performs well under those circumstances.
Keng He: For sure. I mean if I'm a candidate on the other side of the table. I would feel like, 'Hey, the interview session is a collaborative session, right?"
It's more like working with a future colleague rather than, "Oh, I'm being judged all the way throughout the process..."
Keng He: So speaking of the interviewer panels, I'm curious that, do you have like one panel per hiring team or do you have like a shared panel across the entire org or a team?
Catherine, maybe you want to go first?
Cat Miller: Yeah. I mean, we do centralized hiring and I feel pretty strongly that there's a lot of benefits to it. So what we have is a shared interview resource across the entire engineering org. So across all sort of 300 roles that we have. And then we have senior leaders who are designated as hiring managers, but again, they're mostly hiring into a general pool of individuals that we will then match to teams.
This is a little bit different for particular senior roles. So if I'm hiring a director, I might actually be the hiring manager for it. But I can still have, I don't necessarily have only my team doing that interview interviewing and I would actually potentially leverage other hiring managers.
It has an obvious downside that, you know, if you're hiring someone to a general pool, you maybe have less ability to advocate for, "oh, this is a T-shaped candidate, but I think they'll work really well on my team."
So it does require everyone in that position to be aware of what teams exist at the company and what kind of strengths and weaknesses might play in those different spots. You know, there might be some teams that really do well with systems interviewer... people who are really good at systems.
Other teams might be really good with data engineers. So there is some need to do that, but it allows so much standardization. And also means that the teams that are on fire aren't asked to then take on the extra load of getting themselves not on fire. Which I think is a really important benefit.
Keng He: So I do have a follow-up question there, like, in terms of the matching process. Right? And we all know that time to hire is like a measurement, like the very important like that somehow candidates only in the market for like two to four weeks. Right. Once those the times gone the candidates come from the market. And adding a team matching process in between definitely adds the time there.
So is there any tips of how you can manage that process to make it very efficient?
Cat Miller: Yeah. Well for us, we extend the offer simultaneously with team match and the team match can potentially happen after they arrive at the company. So it's a little bit candidate preference.
So for very senior engineers who have a very particular skill set, they might need to match before they joined. But otherwise we actually do a post hiring team match process. And this is something you can get away with when you have a lot of open head count and less so when you have very little open head count.
Keng He: I see. Vinithra, what about the panel on your end
Vinithra Varadharajan: Yeah. So, at Air Table, the only streams that we have are based on skillset. So full-stack front-end or back-end. And so your interview panels themselves will be based off of that. The matching happens at the start. So right from the sourcing, we try and decide which team is a candidate interested in. And the team is also is a good fit. This is based on like our priority of recs.
But this isn't a tight match right at the beginning. Over the course of the interviews, especially sometimes, you know, there are a few recs all those recs get filled out by the time they interview, like the onsite is done. So either for that reason or the candidates preference changes over time, we can revisit the team matching there after.
And so we do have a good system in place that we have understandings with the different groups and the recruiters involved to be able to do these quick... They understand when a team has openings and they can move people around pretty quickly.
So to your point, yes, we try not to spend any time as part of that.
Keng He: I see. So it seems like in this case, the candidate have some indications or hints on which team he, or she will be on. Right. And in Catherine's case, it's just like, if I were a candidate, I have no idea which team I would go into, but I know that I'm going it's going to be like one of the engineering teams, like within Flat Iron Health.
And that only until like, oh, I passed the interview, then I can pick and choose.
Cat Miller: Yeah. And I have to say like, things change really fast. So an opening that exists the day you started interviewing may not exist the day your offers extended. And so that, that is part of the equation here too, is that you know, when things change rapidly, you need to be able to adapt, rapidly.
Keng He: Got it. Got it.
So what are some of the key metrics that you are looking at, or you keep track off to measure the efficiency of the hiring process that you have. Vinithra
Vinithra Varadharajan: Yeah. Oh, so I'd say, you know, like with anything to do with measurement, the sooner you start the better. So for example, time in every stage of the pipeline. And so this is so that we can keep improving our own efficiency. Diversity stats again at every stage of the pipeline. Interviewer scores themselves so that we can see how, like what's the average score that an interviewer gives. And is that higher or lower compared to other interviewers? You can do this even by question. And so this also enables you to give quantitative feedback to different interviewers.
Keng He: Got it. Catherine, what about Flat Iron Health?
Cat Miller: Yeah. I mean, that all sounds right. I think just to elaborate on the sort of the funnel metrics... totally agree about looking at diversity at every stage. And one of the things you're looking here for here is just the drop-off between every stage, so you want to see volume, but you also want to see drop off and you want to see drop off by sort of, demographic factors as well.
And that's an important indicator.
Keng He: So sometimes, like we focus a lot on the engineering side of a house. Right. And then our partners are also recruiting. Right. And do you have any thoughts or like, like, kind of ratios on like, Hey, how many recruiters do I need, right. Or like, how many sources or coordinators, like, do I need in the organization?
Like, maybe there's a magic ratio between a recruiter and a number of openings. I'm just curious, like how you think about this.
Cat Miller: Well, all recruiters are not created equal. I think that this is actually the thing that you're trying to adjust for with your metrics. If you see that you are unable to bring in to the top of your funnel, then you probably need more sourcers. If you find people lingering in different stages of the funnel, then it might mean either that you don't have enough capacity to interview or that you don't have enough basically candidate lead support in order to be able to schedule interviews. If you're losing a lot of people at the end, you know, maybe you don't have enough recruiters to close, or maybe there's something wrong with your process.
So I don't think there's a magic ratio because people are different and situations are different. But I do think this is where your metrics can help you.
Vinithra Varadharajan: Yep. I'd say something similar. Also that to a certain extent it depends on the variety of profiles you're looking for.
So in this quarter, for example, our focus is senior full stack engineers. So then it's that one sourcer can just stay dedicated on that profile and generate a whole bunch of leads.
Versus if I need both backend and full stack on different levels, then that just makes the matrix more complex. And then they're going to spread themselves thin. So you might want to just spread it across two sources.
How has remote work changed your hiring process?
Keng He: So, I mean, in the remote world, right? How did remote changes, like, like everything that we just talked about here in terms of the process, that efficiency and all that, how does it impact.
Cat Miller: I love the remote world. I think that when it started, we were all very trepidatious. Like how we'll be able to evaluate people. Cause we are not a fully remote company. And we weren't even particularly hybrid before this. And I think there was a lot of worry. How will I evaluate people?
There are some interviews that frankly do still suck. When you want to be able to whiteboard a design diagram and you don't have the right tools, that can be really hard.
But there's a lot of advantages. I've loved, you know, you mentioned the being able to spread something over two days instead of one day. The the ability to not have to have candidates fly in. You know, I think that there's actually been a ton of advantages. In fact, you know, people getting sick and being able to swap them out easily, like it's, I'm actually trying currently trying to figure out how we, how much of this we can keep as we go back to in-person.
Vinithra Varadharajan: Yeah. we put together some prep docs for the candidates themselves to help them, because while we might have gotten the practice by now of interviewing people remotely, they might not yet be there. So it's the more of the, what to expect so to speak. And also of course, always audio visual issues like technical issues, you know, what do you do when these things happen? So just having backups and helping them prep ahead of time.
Keng He: Cool. And I think we are at time and then we have a couple of questions coming in from the audience. And thank you so much. Catherine and Vinithra is a great conversation. I learned quite a lot from this Patrick.
Patrick Gallagher: Yeah, and incredible conversation and we had a few questions come in from the audience. The first one who owns the top of the funnel, is it the hiring manager or the people team? And this comes from Rajesh, Agarwal
Vinithra Varadharajan: So in our case we... recruiting does own top of the funnel. However, there is a, for every new job rec that'd be put out a job description that we put out, we do have a calibration session you know, a discussion between the sourcers and the hiring manager to really get precise on the kind of profile that we're looking.
Patrick Gallagher: Is that the same for you, Cat?
Cat Miller: Yeah, I'd say agreed. But then I'll also say that the hiring manager is somewhat responsible for holding recruiting accountable. So there's a give and take there, but definitely recruiting owns filling the funnel..
Vinithra Varadharajan: And I'll add to that, to say that one of the biggest success that we've gotten, like really be, been able to scale is to give that feedback, like we have really tight feedback loops with your recruiters and sources and build, a really trusting relationship. Across that.
Keng He: Yeah, I second that I do think that trust and the collaboration between the recruiting team and hiring managers that are critical to this.
For example, at the top of funnel, I would like sometimes the hiring managers would need to jumping in and look at, "Hey, how did you actually write that email that gets sent out to the candidates, right?"
If we tweak a few words here and there and then make it more technical, maybe it's more appealing to the engineering like audience, right? So it is a collaboration between posts.
Patrick Gallagher: Quick follow up question about the mechanisms of those feedback loops... is to, are you, is it regular meetings? I guess what's the mechanism that you introduced to, to keep that consistent?
Vinithra Varadharajan: Yeah, we do have regular syncs. and, I found that most tactical things can be discussed asynchronously actually to say like, Hey, what do you think about this candidate?
And so on, the more interesting discussions are one observations of what sources and recruiters are observing about the market. And so that we can react quickly to it.
Secondly, is inefficiencies. I think like a bulk of the discussions we've been having lately is like, can we tweak this? Can we tweak that? And it's always a dialogue.
Cat Miller: Yeah Yeah. huge plus one to that. And I'd say we also do QBR's that do focus around those metrics and those funnels so that, you know, you've got the kind of regular touch points for the, like, how can we sort of incrementally improve? And then you have these occasional big picture snapshots of like, are we where we need to be? And what do we have to change?
Vinithra Varadharajan: I think like the name of the game is predictability. And so, especially when it comes to metrics and scale, because everything is a month or two delayed out. Like any improvements you make now, it's going to take some time to manifest.
So being able to predict what your end results are and scaling back from that to make the improvement ahead of time.
Patrick Gallagher: That's so true that the long arc of the feedback loop there.
One more quick question, Cat this is more specific to some of the processes that you were sharing specific to Flat Iron. How do you mitigate a candidate who may be feel like? So I think like the person is like, if somebody's priority is the manager that's important to them as part of the decision making criteria.
How do you mitigate maybe that consideration through your process? Is that like an objection or thing that you deal with ahead of time? I think somebody was curious to know, like, if that's an important consideration and maybe, and that shifts throughout the process, how do you manage that concern sometimes?
Cat Miller: Yeah. I mean, interestingly, that's actually one that I haven't heard a lot. What I hear more is, especially with senior candidates, like, Hey, I really have this technical skill and I want to make sure that there's a project that's appropriate for me.
So we do, to some extent, like we try to suss that out early in terms of the candidates and understand. A lot of candidates actually love the idea that they get to kind of shop around. So it's actually a selling point in many cases.
But for candidates who really want to be like, sort of put somewhere, then we'll do that selling process rapidly at the end, once we've extended an offer, we'll do a rapid couple of reverses. We'll have them pick and go from there. And typically that only adds, I mean, it's a couple of days turnaround on that. It's usually not that that onerous.
Patrick Gallagher: Okay. So it's still sounds like there's a little bit of like that calibration conversation at the end to make sure that it's a good fit for them and making that work.
Cat Miller: Yeah. I mean, if that's the thing they need to know in order to be able to say yes, then like it behooves us to give them the options, have them meet the managers and make their own decision about, you know, whether this is the right place for them.
Patrick Gallagher: Great. Thank you. Well, that is all the time that we have for questions. King vinifera Kat, thank you three so much for an incredible conversation and sharing your insights here. I know that has been really effective for our community, so thank you three.
Cat Miller: Thank you.
Keng He: Thank you.
Vinithra Varadharajan: Thank you. This was great!