Post by Cory Doctorow

@pluralistic One talk, that was quite interesting, was from a company with 180 developers that did a slow and careful rollout of AI, mainly due to high demand. They did a self-assessment, and developers claimed increased productivity. They said it was too early to say if it was actually true though, and promised to come back with results (presumably another year) if they had any.

It's the same where I work. Developers *feel* more productive, but I have yet to see any numbers backing it up.

@pluralistic At the tech conference I was at last week, which was of course 2/3 AI, it was often taken as self-evident that AI had given us huge gains. I should look up some of those people and send them this ...

@veronica You can probably anticipate their reactions as much as I do. It's pointless. AI is a waste of time. Even warning of AI is a waste of time.

@burningTyger There were several good talks on AI there too, so a good chunk of the audience got a balanced message overall. I also attended a few talks on AI usage, but only those directly relevant to my work (they did after all pay for my ticket).

@veronica @pluralistic I think they are more productive, given a definition of 'productive' that measures output.

You can definitely produce much more... 'stuff'. Some of it might even be useful. A lot of it turns out to waste other people's time. But as long as you have a definition of productivity that only measures output and not bottom line results, it's a useless metric.

@loke Indeed. If your metric is output, then sure. It's the age old "quality or quantity" conundrum. If it's not useful output, or you need to frequently replace it with new generated code, is it then productive in a meaningful sense? Or, in my opinion, worse: Is it just increased complexity and fragility for no or minimal gain?

@pluralistic

@veronica @pluralistic I have looked at two AI tools, one is a simple line completion model that guesses at what I'm about to type. It's a simple model that runs locally and is right about 80% of the time. Overall I think it offers a benefit.

The other is a big model that tries to figure out the next few lines, and even after letting it at a code base for some time, the guesses are wildly inaccurate, and mess with keystrokes, so I wind up having that inaccurate code injected, which means I then need to remove it before typing what I was already planning to type. This has lowered my productivity. I have used this model to generate test cases; about 80% of the cases made sense, the rest needed minor tweaks, so there's a potential use case there.

I will try getting an advanced model to build a small application, but at this point I expect I'll spend as much time fixing the generated code as it would have taken to write it in the first place.

One place where it might be useful is generating code in a language I am familiar with but haven't mastered. At that point the logic would still be auditable, and I could use that review to raise my level of expertise.

@alan I find the VSCode auto-complete feature highly annoying and I mostly just keep it off. Some of the stuff I work on is pretty obscure, like parsing streaming binary subtitle data from DVD/Bluray, and it keeps suggesting code that is obviously stolen from somewhere else (and also doesn't fit my use case). I've also had it echo back my own code too, when it's relatively rare stuff. Funnily, it came with a license warning last time. I cannot recall them asking to ingest my code without credit!