I've worked at a place with a heavy "buy" emphasis and can explain how it happened. AI may change this, for sure.
Yes, many of the engineers including me would much rather build our own thing and make it tailored to what we need. Two engineers is enough to build a lot of the things you would buy, and five would be plenty.
One of the problems is that there is plenty of empire for a manager who goes with Buy. You get a large budget and get to have your choice from the market of who will implement it. From a lifestyle perspective, it is also more relaxing. Vendors have to compete to keep your favor and can be replaced, so they will contribute lots of workers who go look through your systems and figure out how to integrate with you. To contrast, internal workers have long learned to game the systems and will do the least possible measurable work while finding time to do side projects that help their own futures, e.g. unnecessary internal libraries and interest groups. I was given overwhelming pressure from the management chain to come up with reasons to "buy" something.
"Buy" is also often a revolution rather than an evolution, and the lovely thing about a revolution project is that you get 1-2 years of low accountability while you get the integration started. I saw this over and over with Buy decisions that either never really worked or that technically worked but required large internal teams for the integration due to it being designed badly or not being a great fit.
Another issue with Build is that there is usually an incumbent team already that will fight to keep their project going. With Buy, it is much easier to play the vendors off against each other since they are all outsiders. With Build, if the first version isn't done right, it can be nearly impossible to fund a second Build for the same thing or to replace the incumbent staff or to demand a new design.
One of the biggest issues is that it's hard to find and assign the right worker for a Build project. You need someone with enough breadth in computer science to make all the little decisions that will make an internal tool work well. It's very different from the main software engineering track, which converts large quanities of requirements from the product managers into running code. It's hard to find and identify the right people, and you will face a lot of pressure to use your best external-product people since that is 90% of the software group's output and will have lots of supporting processes and norms. Lots of people will raise their hands for these teams, and it will be a trap if you select your ones that are great at external products.
Google used to do this well, though I expect they are less good at it nowadays. Think about systems like protocol buffers, Map Reduce, or Spanner. At most companies, these will not even get funded if the decision is build versus buy, because they are new tools that won't show up in the internal customer surveys a PM will do. If they do get funded, think of the challenges to staff the core design role. On top of all that, until the tool actually works and becomes popular, consider what your budget it--it's 1-2 system designers and maybe 2-4 additional engineers to build stuff. The reason Google built all of these is that they identified the right person--literally one person for all of those--and gave them room to do what they do. That's unusual to pull off.
I feel like larger companies just move toward Buy to a large degree. I agree it can be miserable to sell to them if you try it as a random shmuck, but it is not so tough if you use professional sales people. Sales people are experienced at analyzing a target company, learning how it works, and then cracking the puzzle with a sequence of calls, meetings, and structured decisions to get from "no" to "yes". The individual people at the target company are looking out for themselves at a larger company, so they are amenable to this puzzle-solving and will even help you do it if you approach them the right way.
Building rather than buying isn’t just about empire building. Third party systems are often built around data models and interfaces that aren’t compatible with existing internal systems. This forces the enterprise to either migrate their existing systems to the new model or build a complex integration layer. Almost all enterprise software has to be customized, sometimes heavily. If the Enterprise already has software engineers, it’s often easier and faster to build than buy. Buying has long term costs that are never reflected in the initial proposal. These systems often become unsupported, legacy stacks and a major source of technical debt.
I agree about the company's POV, but the incentives for a middle manager can be to throw bodies at the integration work, which is something easy and predictable to map to dollars and cents. Everyone comes out ahead, and even the company itself still does okay, just is way less efficient than it could be.
I had experience with two prototype web sites around 1997 on my 486DX PC running Linux. The first would have really benefited from AI. A retired guy had an idea for a complaint website; memory says it was a four step process: choose company, product, and model, then write up your complaint. Monetization would be from selling messaging access to the manufacturers, a semi-extortionate process, but still honest, I think. I did the backend and database, a friend handled the graphics and layout. We made it really really clear to him, or at least tried to, that we were only interested in the prototype proof of concept that he could shop around to "real" developers, that we knew nothing about scaling up for production. He claimed he had a million bucks in backers. Turned out to be a lie; he was raiding his retirement fund, and once his wife found out, she put a stop to it. He then decided he wanted to change the entire model from that four step process, that we should do the work free as a bug fix. It was working and running on my 486DX, and after a couple of months of no further accesses, I pulled the plug.
The second one was much more straightforward. An experienced businessman wanted a web site for coupons to replace those newspaper clippings I still get in the mail. The idea was that office mates are going to lunch; where to go? Visit the coupon site! See who has specials, print the coupons, and they include the kind of detailed info which you can't get from mass market newspaper coupons: weather, time of day, location (zip code? I forget). He knew exactly what he wanted, he had the graphics and layout, and I wrote the prototype backend, again running on my 486DX. He actually shopped it around for six months or so, and I personally thought he had a good project. But in spite of his successful business background (he owned a $2.7 million house at Tahoe and had sold his previous startup), the Internet was too new and he couldn't interest anyone in turning it into a real public website. I don't think AI would have helped at all, since the prototype was simple and working, he just couldn't sell it in those early days.
I've worked at a place with a heavy "buy" emphasis and can explain how it happened. AI may change this, for sure.
Yes, many of the engineers including me would much rather build our own thing and make it tailored to what we need. Two engineers is enough to build a lot of the things you would buy, and five would be plenty.
One of the problems is that there is plenty of empire for a manager who goes with Buy. You get a large budget and get to have your choice from the market of who will implement it. From a lifestyle perspective, it is also more relaxing. Vendors have to compete to keep your favor and can be replaced, so they will contribute lots of workers who go look through your systems and figure out how to integrate with you. To contrast, internal workers have long learned to game the systems and will do the least possible measurable work while finding time to do side projects that help their own futures, e.g. unnecessary internal libraries and interest groups. I was given overwhelming pressure from the management chain to come up with reasons to "buy" something.
"Buy" is also often a revolution rather than an evolution, and the lovely thing about a revolution project is that you get 1-2 years of low accountability while you get the integration started. I saw this over and over with Buy decisions that either never really worked or that technically worked but required large internal teams for the integration due to it being designed badly or not being a great fit.
Another issue with Build is that there is usually an incumbent team already that will fight to keep their project going. With Buy, it is much easier to play the vendors off against each other since they are all outsiders. With Build, if the first version isn't done right, it can be nearly impossible to fund a second Build for the same thing or to replace the incumbent staff or to demand a new design.
One of the biggest issues is that it's hard to find and assign the right worker for a Build project. You need someone with enough breadth in computer science to make all the little decisions that will make an internal tool work well. It's very different from the main software engineering track, which converts large quanities of requirements from the product managers into running code. It's hard to find and identify the right people, and you will face a lot of pressure to use your best external-product people since that is 90% of the software group's output and will have lots of supporting processes and norms. Lots of people will raise their hands for these teams, and it will be a trap if you select your ones that are great at external products.
Google used to do this well, though I expect they are less good at it nowadays. Think about systems like protocol buffers, Map Reduce, or Spanner. At most companies, these will not even get funded if the decision is build versus buy, because they are new tools that won't show up in the internal customer surveys a PM will do. If they do get funded, think of the challenges to staff the core design role. On top of all that, until the tool actually works and becomes popular, consider what your budget it--it's 1-2 system designers and maybe 2-4 additional engineers to build stuff. The reason Google built all of these is that they identified the right person--literally one person for all of those--and gave them room to do what they do. That's unusual to pull off.
I feel like larger companies just move toward Buy to a large degree. I agree it can be miserable to sell to them if you try it as a random shmuck, but it is not so tough if you use professional sales people. Sales people are experienced at analyzing a target company, learning how it works, and then cracking the puzzle with a sequence of calls, meetings, and structured decisions to get from "no" to "yes". The individual people at the target company are looking out for themselves at a larger company, so they are amenable to this puzzle-solving and will even help you do it if you approach them the right way.
Building rather than buying isn’t just about empire building. Third party systems are often built around data models and interfaces that aren’t compatible with existing internal systems. This forces the enterprise to either migrate their existing systems to the new model or build a complex integration layer. Almost all enterprise software has to be customized, sometimes heavily. If the Enterprise already has software engineers, it’s often easier and faster to build than buy. Buying has long term costs that are never reflected in the initial proposal. These systems often become unsupported, legacy stacks and a major source of technical debt.
I agree about the company's POV, but the incentives for a middle manager can be to throw bodies at the integration work, which is something easy and predictable to map to dollars and cents. Everyone comes out ahead, and even the company itself still does okay, just is way less efficient than it could be.
I had experience with two prototype web sites around 1997 on my 486DX PC running Linux. The first would have really benefited from AI. A retired guy had an idea for a complaint website; memory says it was a four step process: choose company, product, and model, then write up your complaint. Monetization would be from selling messaging access to the manufacturers, a semi-extortionate process, but still honest, I think. I did the backend and database, a friend handled the graphics and layout. We made it really really clear to him, or at least tried to, that we were only interested in the prototype proof of concept that he could shop around to "real" developers, that we knew nothing about scaling up for production. He claimed he had a million bucks in backers. Turned out to be a lie; he was raiding his retirement fund, and once his wife found out, she put a stop to it. He then decided he wanted to change the entire model from that four step process, that we should do the work free as a bug fix. It was working and running on my 486DX, and after a couple of months of no further accesses, I pulled the plug.
The second one was much more straightforward. An experienced businessman wanted a web site for coupons to replace those newspaper clippings I still get in the mail. The idea was that office mates are going to lunch; where to go? Visit the coupon site! See who has specials, print the coupons, and they include the kind of detailed info which you can't get from mass market newspaper coupons: weather, time of day, location (zip code? I forget). He knew exactly what he wanted, he had the graphics and layout, and I wrote the prototype backend, again running on my 486DX. He actually shopped it around for six months or so, and I personally thought he had a good project. But in spite of his successful business background (he owned a $2.7 million house at Tahoe and had sold his previous startup), the Internet was too new and he couldn't interest anyone in turning it into a real public website. I don't think AI would have helped at all, since the prototype was simple and working, he just couldn't sell it in those early days.