Would I hire AI as a software engineer? Job interview with ChatGPT.
ChatGPT is hot right now! People are trying different things, asking weird questions, posing challenging topics, and all that to understand how bright the ChatGPT is. It’s a search for ways to skew the AI’s logic to say something awful, controversial or scary. Honestly, that’s one of the main reasons it got so much attention recently. Humans are driven by fear, and we want to find the limits by asking things like, “How can I get away with a murder?”. So you can see how AI’s valid-looking response to a question like that would get the whole world’s attention, and probably resulting in the immediate shutting down of the app.
I decided to use it for a slightly different purpose and see if we could hire ChatGPT as a software engineer, even in a junior position. I was hoping to reach a conclusion to a question…
Can AI compete with humans with all these impressive technical answers and disrupt the software engineering world, which fears that it might end many software careers?
Interview process
Our interview process is straightforward. Two stages and a mix of techy and “soft” questions about the interviewee. We don’t do homework and peer-programming exercises. Instead, it’s all about the candidate and understanding how they fit in our company, where they could bring value and how this role can help their careers as a software engineer.
Screening process
I did ask the AI a few typical technical screening questions, and I wasn’t surprised at all that it came back with almost the best answers I’ve seen. Although they were accurate, they all missed a bit of uncertainty or “experience added value”, which often comes up during a conversation with a human being. Dry — is probably the best word to describe them.
Because it’s an AI, I couldn’t ask about previous experience, skills, knowledge, or reasons why it’s looking for a new job. Believe me, I tried. Unfortunately, I had to skip all the typical questions that help me understand the person I’m talking to.
This candidate (the AI) would not pass the interview process to the next stage. Technical knowledge is one of the easiest things to learn (or fake). However, applying that knowledge and real-world experience is what matters, and AI can’t do it (at least not yet).
Main interview
Unsurprisingly, I ended up with similar results during the second phase as I had during the screening process. Again, no emotion-based questions were answered, which would disqualify the candidate before we even go to the tech exercises.
How about the tech interview?
ChatGPT was very good in code comprehension reading, giving very accurate and satisfactory answers, but it failed two technical tests. One was related to JavaScript code — AI explained what would happen and why, but the explanation was wrong, which resulted in an incorrect final response. The other one was related to CSS, and it’s a straightforward one which should be an easy task for a data-driven machine. All the technical explanations provided by AI suggested a lot of “book knowledge” and possibly experience writing algorithms and complex code but almost no real-world experience with user-facing apps. So we are again seeing “no real-world application of the knowledge”.
Is AI more human than I expected?
This led me to do another test where I realised AI sometimes has no clue what it’s talking about, as it gave me three different answers to the same question. One of these answers was actually correct. The AI is more human than I expected, changing its mind or just trying to BS me.
Homework tasks are usuless!
Typical homework tests where a candidate is asked to write something specific, do a task in their spare time and send the code back became even more useless. I’ve always found these kinds of projects valueless from a hiring perspective, nor I enjoyed them myself when looking for a job years ago. They are all just big waste of time with no strong correlation between results and a person’s technical skills. To make it worse, candidates can now use ChatGPT to do the job for them in a fraction of the time and not bother friends to help them.
Will the result be suitable for a senior role?
I doubt it. But it is likely good enough for a junior position.
Would I hire AI as a software engineer?
There is a straightforward answer to this question: NO.
Many companies where tech test is the essence of the interview would be very close to bringing someone like that on board. Of course, it’s doubtful that a candidate will be hired based only on technical skills. Still, with a poorly designed interview, I see how a non-engineer can game the process and get a highly-paid job with barely any experience or technical knowledge.
Why this suddenly sounds like every politician’s job interview process? :)
Breakthrough technology?
- Just like jQuery, introduced over 10 years ago, made JavaScript easier and accessible to a broader audience.
- Just like React made building complex UI apps possible for less experienced devs.
- AI will lower the entry bar for many non-technical persons to write meaningful and “good enough” code which fits into the modern NoCode trend.
All these revolutions have something in common — people who jumped straight into them without fundamental knowledge of the topic can build stuff quickly but have no understanding of why or how things work. It creates a tendency where complex software solutions are very hard to understand, not because they are complex but because they were written by low-grade programmers. I believe the AI solutions like chatGPT will add to this challenge. It will likely replace StackOverflow for many daily software tasks reasonably soon and become a tool to help solve problems faster, but only sometimes better. There are always two sides to a medal.
Whilst everybody is excited or even fears the new chat, it should be barely any news for programmers. GitHub has been in this space for over a year with its purpose-built product called Co-Pilot. In terms of coding, it’s doing a much better job than ChatGPT, but it’s not as accessible as typing a question in a browser and getting an answer.
I will write about Co-Pilot in my next blog post.
Worried about software engineering careers?
When AI gains real emotions, that’s the moment when we might start worrying about programmers’ jobs. The question is — will it ever happen? And if it will, how hard will it be to verify that (because it will also learn deception)? Emotions are needed to make meaningful decisions that serve ourselves and/or others; otherwise, they are only data-driven choices which might bring destructive results.
One thing to remember…
We can be almost sure that AI cannot be (at this stage) interviewed like a human and will likely fail to get a job. But AI is not human! If we can see its value as a tool and harness the power behind it… That’s a game changer and already creeping into the software engineering world.
