The Unhatched Cost of AI Reliance on Engineering

Two days ago, I went to the local BDC dealer to exchange 100 dollars to naira. After we had agreed on the exchange rate, I handed him over the 100-dollar bill. The first thing he did was raise it up in the direction of sunlight, then looked at me straight in the eye, then pocketed the note.

I was left wondering if he thought I must have facilitated it through a rogue means because of course, it looked very neat and clean.

This was his own local means of determining the authenticity of a note. It was very easy for him to determine if the note I had given to him was an original or a counterfeit. This is not a one-off event; it is a result of the knowledge accrued from years of interacting with several dollar notes, and as a result he can easily determine if a note is a counterfeit or not.

How do we determine if the output, performance or benchmark set by an AI agent is correct?

There has been a lot of speculations whether software engineers should now read AI generated code to verify its correctness and intent or, if it generally works, then let us hope and pray. I quite do not agree to this philosophy because it only sets a dangerous precedent for the underlying harm which would only take years before it is noticed; but cooperate tech giants doesn't want you to look pass this direction because it is a win-win for them regardless.

AI has become so good at code generation which is a significant development to how we now write software. Currently, AI can write code as good as a senior engineer can, surprisingly, it can also spot and correct badly written code. This should be the peak of productivity as a software engineer.

But there is one problem which AI still doesn't seem to understand, context.

The issue of code generation goes far beyond syntax and logic generation. It is in fact the cumulated understanding of the scope of which the business logic lies. This drives us back into the human factor. All software is written for human consumption, even for automated system or machine-to-machine integrations, the final output has to do something which humans desire to achieve, this in par means humans have to write concise specifications on what they intend to achieve in details for the machine to understand.

You see, I've decided to scratch the basics of context because context when it comes to engineering requirements slightly differs from that of business requirements. Requirements such as performance, reproducibility, extensibility, integration, several others which serve as a chain to the other for delivering a trustworthy solution.

When Andres Freund discovered a massive nation-state cybersecurity backdoor (later named the xz backdoor) in SSH of a particular linux distribution (which was later discovered to affect all distributions is deployed), this wasn't any luck or coincidence. This was a result of years of professional experience and understanding of the fundamentals of how the system should look like and work. This includes years of professional development, debugging, benchmarking and finetuning. How would one know these things in today's software development era?

A 500ms delay which Andres found in xz utils is such an insignificant timespan in most applications of software, and in fact, people almost always dismiss it when I mention a 1sec lag in execution of an event in end-user applications, because they see it as insignificant.

It is a disturbing trend today which leadership of big tech sell you the idea of AI reliance. It is of no doubt that AI is very smart considering the level of knowledge it is trained upon, but total reliance on AI to determine a benchmark, what is attainable, right, and conceived is the foundation to a generational system collapse which they are incubating.

Furthermore, who benefits when everyone becomes bad engineers? I guess you should know where this is heading to now. The very companies which sell you the dependability notion becomes the sole actor in the system who determines what is right, attainable, the terms which these can be achieved, infrastructural cost and lots of unpredictable outcomes.

AI is a very important tool in today's workspace, it has transformed almost everything we do and given us the abilities to do the things we never thought of achieving, but total reliance on AI agents and systems to determine how our workspaces, ecosystem and future should be is only but a hatching trap we are walking straight into.