Here are some interesting points:
1. Code analysis firm sees no major benefits from AI dev tool when measuring key programming metrics.
2. …found no significant improvements for developers using Copilot.
Hoffman acknowledges there may be more ways to measure developer productivity than PR cycle time and PR throughput, but Uplevel sees those metrics as a solid measure of developer output.
> “It becomes increasingly more challenging to understand and debug the AI-generated code, and troubleshooting becomes so resource-intensive that it is easier to rewrite the code from scratch than fix it.”
—Ivan Gekht, CEO, Gehtsoft
Final analysis with positive caveat:
>“Expectations around coding assistants should be tempered because they won’t write all the code or even all the correct code on the first attempt,” he says. “It is an iterative process that, when used correctly, enables a developer to increase the speed of their coding by two or three times.”
thomascgalvin on
I think AI can be very useful when you’re using it like Stack Overflow plus plus; if you ask it *how do I connect to a datasource in Spring Boot* or *how to I create a channel in GoLang* it can usually spit out a decent answer. You still need to verify and test, but it’s usually correct, or close to correct.
But it falls apart when you start asking it more open-ended questions. If you don’t already know *why* you need a datasource or a channel, you can’t ask the AI the right questions to make it give you the right answer.
If you treat AI like documentation, you’re fine. If you treat it like an architect, you’re out of luck.
mr_claudy on
I love Copilot as a Neovim completion source. Though in control, I type half as much as before.
thatfreshjive on
It’s useful for boilerplate or starting a project, but it can’t handle the complexity of modifying a large code base
-not_a_knife on
I’m not a dev by any means but I find it useful as an entry point for learn a new subject. I find it terrible to rely on for anything that requires nuance, though. For instance, I was trying to set up a Qemu VM and I talked in circles with it about networking. Also, a few days ago I tried to auto generate a very basic bash script and it would completely ignore parts of my prompt. Both of these situations lead me to learn a lot but I don’t think that’s what these AI companies are selling.
Oprofessorfps on
I save so much time in boilerplate. A good workflow boosts efficiency.
phdoofus on
I’ve been programming for a long time now but it’s a tool and a job for me not a lifestyle (my background is actually in earth science) so there are a number of things i’ve never learned to do and can’t be bothered so for me it’s a useful tool to say ‘hey write me a bit of code in <language> that does X’ and it gives me something and I look it over and bam. Task done. I don’t have to go out and waste a lot of time getting good at a language or something I don’t care about learning. Sometimes i ask it to explain something given a code example and it usually does a credible job but then it can also get stuck and give you something basically with zero value.
Jelkec on
Is it unusual that I don’t use AI in my workflow? Starting to feel like it.
AI isn’t useful in software development. While many are desperately trying to figure out its utility, I don’t see it.
Ffs, I’ve seen ads for an AI-first pull request review system. Why would I want that? Do we trust LLMs more than software developers?
__Loot__ on
Its far from perfect but I have not had to go stack overflow for 2 years accept 1 time. And I asked for help on reddit and discord 2 times each. Before Ai it was every 3days or less but I was pretty new to programming back then. It’s life changing for me because I had a stroke 7 years ago so I was programming at 7 to 9 words per min. Its like a good junior level programmer
jarkon-anderslammer on
The Devs that gain little from it are the same Devs that are going to be out of work because of it.
joshthor on
Ai is great for coding when you keep the asks short and sweet. If there are more than 2-3 processing steps it starts to show weakness though. But ai is way more convenient than documentation or digging though stack overflow if you are building with a new library or language and don’t know the syntax.
vivaciousnexus on
AI assistants can be pretty handy for quick syntax checks or generating boilerplate code. but you’re right, they’re not replacing actual dev skills anytime soon. good devs know how to architect solutions and debug complex issues – that’s where the real value is. AI’s just another tool in the toolbox, not a silver bullet.
Master_Engineering_9 on
devs trying to justify their existence.
SHODAN117 on
Tell that to useless busy body managers or C-suits. They salivate at the thought of replacing you and think we’re just around the corner to that.
Necessary_Petals on
What are the layoffs about? They need 50% less coders, but, maybe it’s to make room for new coders!
CoherentPanda on
Bullshit article
sicbot on
I have a co worker who really likes AI to help him code…but all he uses it for is to basically ask stack overflow questions (gee I wonder were the AI got the answer from…) or to get starter templates for different functions…something he could do with a snippet library.
In the last month I’ve asked him for help twice and he gave me AI answers both times and they were both very wrong lol.
thaldin_nb on
It’s another tool in our toolbox. It is not a replacement for writing code. Though I admit I find it useful for speed creating unit tests.
positivitittie on
Anyone using any off the shelf AI coding assistant will have these results.
With an embarrassingly small amount of work you can get AI to do so much more.
I been coding for 30 years and I just let the LLMs do it anymore. To me, it became boring the moment I saw an AI do my job. But it’s still fun to create and I control that.
tetsuo_7w on
Copilot keeps telling me to use library functions that don’t exist. When confronted, it apologizes and tells me to use another library function that doesn’t exist. Rinse and repeat until it returns to the first function that doesn’t exist and the loop is complete.
imselfinnit on
I’m still going to imagine the little mouse from the animated movie Ratatouille, hiding in the cook’s hat, directing the action.
badmattwa on
The author of this article is out of their element, and does not seem to understand the reality of the subject
Zugas on
AI is useless.
AI is too powerful.
Which one is it?
ShankThatSnitch on
The most useful thing I’ve used it for is dropping code from an unfamiliar language into it, and it spits out nice and concise descriptions about what the various syntax, operators, attributes…etc are.
For instance, plopping in a random line of REGEX, and it tells you what each part is for and how to use it. That is very useful.
Embarrassed_Quit_450 on
That’s a rather stark contrast with the heap of business analysts predicting AI will replace all devs in five years.
gizamo on
I gained both hope and disappointment.
In that order, and repeatedly. Good times.
legshampoo on
saves me hours per day that would be spent scouring forums and blogs for answers to simple question. it summarizes and provides exactly the syntax i need, instead of me having to scan walls of text tor what i’m looking for
petesapai on
Wrong, The CEO of NVIDIA was clear that no one should ever study computer science and coders are obsolete…
If you can’t trust the leader of a company that would throw his mom under a bus if it meant increasing his stock prices, who can you trust… /s
Makabajones on
My work makes us run everything through Microsoft copilot and holy shit is it garbage
30 Comments
Here are some interesting points:
1. Code analysis firm sees no major benefits from AI dev tool when measuring key programming metrics.
2. …found no significant improvements for developers using Copilot.
Hoffman acknowledges there may be more ways to measure developer productivity than PR cycle time and PR throughput, but Uplevel sees those metrics as a solid measure of developer output.
> “It becomes increasingly more challenging to understand and debug the AI-generated code, and troubleshooting becomes so resource-intensive that it is easier to rewrite the code from scratch than fix it.”
—Ivan Gekht, CEO, Gehtsoft
Final analysis with positive caveat:
>“Expectations around coding assistants should be tempered because they won’t write all the code or even all the correct code on the first attempt,” he says. “It is an iterative process that, when used correctly, enables a developer to increase the speed of their coding by two or three times.”
I think AI can be very useful when you’re using it like Stack Overflow plus plus; if you ask it *how do I connect to a datasource in Spring Boot* or *how to I create a channel in GoLang* it can usually spit out a decent answer. You still need to verify and test, but it’s usually correct, or close to correct.
But it falls apart when you start asking it more open-ended questions. If you don’t already know *why* you need a datasource or a channel, you can’t ask the AI the right questions to make it give you the right answer.
If you treat AI like documentation, you’re fine. If you treat it like an architect, you’re out of luck.
I love Copilot as a Neovim completion source. Though in control, I type half as much as before.
It’s useful for boilerplate or starting a project, but it can’t handle the complexity of modifying a large code base
I’m not a dev by any means but I find it useful as an entry point for learn a new subject. I find it terrible to rely on for anything that requires nuance, though. For instance, I was trying to set up a Qemu VM and I talked in circles with it about networking. Also, a few days ago I tried to auto generate a very basic bash script and it would completely ignore parts of my prompt. Both of these situations lead me to learn a lot but I don’t think that’s what these AI companies are selling.
I save so much time in boilerplate. A good workflow boosts efficiency.
I’ve been programming for a long time now but it’s a tool and a job for me not a lifestyle (my background is actually in earth science) so there are a number of things i’ve never learned to do and can’t be bothered so for me it’s a useful tool to say ‘hey write me a bit of code in <language> that does X’ and it gives me something and I look it over and bam. Task done. I don’t have to go out and waste a lot of time getting good at a language or something I don’t care about learning. Sometimes i ask it to explain something given a code example and it usually does a credible job but then it can also get stuck and give you something basically with zero value.
Is it unusual that I don’t use AI in my workflow? Starting to feel like it.
AI isn’t useful in software development. While many are desperately trying to figure out its utility, I don’t see it.
Ffs, I’ve seen ads for an AI-first pull request review system. Why would I want that? Do we trust LLMs more than software developers?
Its far from perfect but I have not had to go stack overflow for 2 years accept 1 time. And I asked for help on reddit and discord 2 times each. Before Ai it was every 3days or less but I was pretty new to programming back then. It’s life changing for me because I had a stroke 7 years ago so I was programming at 7 to 9 words per min. Its like a good junior level programmer
The Devs that gain little from it are the same Devs that are going to be out of work because of it.
Ai is great for coding when you keep the asks short and sweet. If there are more than 2-3 processing steps it starts to show weakness though. But ai is way more convenient than documentation or digging though stack overflow if you are building with a new library or language and don’t know the syntax.
AI assistants can be pretty handy for quick syntax checks or generating boilerplate code. but you’re right, they’re not replacing actual dev skills anytime soon. good devs know how to architect solutions and debug complex issues – that’s where the real value is. AI’s just another tool in the toolbox, not a silver bullet.
devs trying to justify their existence.
Tell that to useless busy body managers or C-suits. They salivate at the thought of replacing you and think we’re just around the corner to that.
What are the layoffs about? They need 50% less coders, but, maybe it’s to make room for new coders!
Bullshit article
I have a co worker who really likes AI to help him code…but all he uses it for is to basically ask stack overflow questions (gee I wonder were the AI got the answer from…) or to get starter templates for different functions…something he could do with a snippet library.
In the last month I’ve asked him for help twice and he gave me AI answers both times and they were both very wrong lol.
It’s another tool in our toolbox. It is not a replacement for writing code. Though I admit I find it useful for speed creating unit tests.
Anyone using any off the shelf AI coding assistant will have these results.
With an embarrassingly small amount of work you can get AI to do so much more.
I been coding for 30 years and I just let the LLMs do it anymore. To me, it became boring the moment I saw an AI do my job. But it’s still fun to create and I control that.
Copilot keeps telling me to use library functions that don’t exist. When confronted, it apologizes and tells me to use another library function that doesn’t exist. Rinse and repeat until it returns to the first function that doesn’t exist and the loop is complete.
I’m still going to imagine the little mouse from the animated movie Ratatouille, hiding in the cook’s hat, directing the action.
The author of this article is out of their element, and does not seem to understand the reality of the subject
AI is useless.
AI is too powerful.
Which one is it?
The most useful thing I’ve used it for is dropping code from an unfamiliar language into it, and it spits out nice and concise descriptions about what the various syntax, operators, attributes…etc are.
For instance, plopping in a random line of REGEX, and it tells you what each part is for and how to use it. That is very useful.
That’s a rather stark contrast with the heap of business analysts predicting AI will replace all devs in five years.
I gained both hope and disappointment.
In that order, and repeatedly. Good times.
saves me hours per day that would be spent scouring forums and blogs for answers to simple question. it summarizes and provides exactly the syntax i need, instead of me having to scan walls of text tor what i’m looking for
Wrong, The CEO of NVIDIA was clear that no one should ever study computer science and coders are obsolete…
If you can’t trust the leader of a company that would throw his mom under a bus if it meant increasing his stock prices, who can you trust… /s
My work makes us run everything through Microsoft copilot and holy shit is it garbage
It lies a lot.