Software development seems to be suffering badly from NIH. There are plenty of useful things that could be developed. But a lot of time seems to be going into duplicating similar software that already exists.
It does seem to be a perennial issue. Long ago, excellent algorithms were published and collected by the ACM (the Association for Computing Machinery). Often they had been carefully researched, optimised and compared with alternatives. Around the same time, Donald Knuth published his classic works on algorithms. Again, validated as working and efficient, these were excellent methods for carrying out common tasks. Yet even people writing operating systems, who you might have thought would be interested in efficiency, implemented inferior algorithms when better ones were publicly available.
Maybe it is because it is more fun to get cracking and write some code that to search out ideas from other people. Or maybe many software developers didn’t see published works as valuable sources of ideas. Or perhaps they just wanted to do things their own way, even if it wasn’t a very good way.
Writing before thinking seems to be one of the downfalls of software development.
Another idea seems to have suffered a slightly different fate. The notion of patterns got explosive publicity. But then the whole idea was misunderstood and misapplied. Developers wrote what they imagined to be a general implementation of a pattern. But something that can be written in a general way is not a pattern — it is an algorithm. The whole idea of a pattern is something that remains abstract and has to be implemented in different ways for each case.
A further problem with patterns has been over engineering. People have been determined to see patterns and force problems to fit them. Often, this has led to excessively elaborate solutions that are inefficient and inflexible. Putting it more simply, it has led to bloat.
I have worked in many areas, from banking to pipeline telemetry. But nowadays, I spend time developing extensions for the Joomla CMS. Probably a system that is in decline, but by no means dead. Again, there seem to be many scattered implementations doing much the same thing, and relatively few generalised systems to aid development.
An example of fragmented development is payment mechanisms. Numerous Joomla extensions cater for user payments. A common approach is to have plugins for different payment systems. Now one would think that it would be desirable to have a single interface between applications and payment plugins. Then there could be competition for the best plugins, offering the best value. A set of payment gateways that is widely used and freely available might be the most popular — such as Omnipay — yet I am not aware of an implementation of Omnipay for Joomla.
Instead of a generalised approach, developers generally write their own plugins to varying standards, even within their own product ranges. Possibly this is a way to make money, as web sites are obliged to pay for plugins, often paying multiple times for the same payment provider, such as Stripe. Perhaps it is a way to make money, but overall it is inefficient and costly.
Communicating information about software is a problem. Not because there isn’t any, but more because there is too much. Reviews of the best software for a given purpose often turn out to be by software vendors — and their own product comes out on top! Rarely do different reviews agree on much. And libraries such as Github have filled up with stalled and broken products.
So I don’t have an answer, other than to urge software developers to try harder to find ways to exploit the best work that has already been done.