«It depends». That may be the most used sentence when evaluating the correct technology for a generic challenge. Choosing the right technology is not a trivial problem. When the needs are poorly defined, or not defined at all, it may not even be possible to suggest a good solution. But even if the problem is broken down into simple technical challenges, there are multiple factors that influence the implementation of the solution. Budget constraints, internal knowledge, timelines, resourcing issues, corporate policies, integration with existing systems, technical feasibility, and a long etcetera will influence the final decision.
Leaving these other—extremely valid—considerations aside, let’s focus on the technical part. Often times you have different frameworks, applications, libraries, SAAS, … to provide a solution with different degrees of success. The sheer size of this list of options can complicate the decision. It is common practice to check what everyone else is doing. Doing market research is a good first step, but it should not determine the final decision.
Many times when a new technology arises, there is media hype. All of a sudden many computer science related blogs and news sites are flooded with posts explaining the merits of this groundbreaking new solution. Often times, these communications are very passionate and contagious, and that’s good! Their goal is to get other people to try the new tools. Information flows rapidly and allows people that are solving that particular problem to assess if they want to try a new direction.
This situation can result in what I call the shiny effect. I admit that I have been blinded by it in the past. I have spent large amounts of time learning frameworks that turned out to be great for only a narrow set of use cases. That has made me cautious about how I approach new tools, even though I still get excited about them.
One widely used argument is based on authority. (If all these multi-milion dollar organizations are using this solution for the exact same problem I have, then I should do the same. After all, they would never choose to implement it without careful analysis.) Resorting to the argument from authority will not always lead you to a reliable answer. The market trend will not always be accurate, there are multiple examples of that.
I remember how in a not so distant past, NoSQL databases were postulated by some people as the future, some people went so far as to say that SQL was dead. There was a time when Apache was to be completely usurped by other alternatives. The LAMP stack was supposed to be replaced by the MEAN stack, without leaving any trace. SOAP and RPC were to disappear because of REST, while REST is now irrelevant because of GraphQL. Also, at the present time no one was supposed to be using Basic Auth. It goes without saying that the opposite happened as well: prophecies of emerging solutions being doomed to disappear that were never fulfilled.
None of that happened. All the tools and technologies listed above have found a way to coexist and share the spectrum of solutions. Even when they seemed to overlap at the beginning, the emerging alternatives ended up being only a better solution for a subset of scenarios in most of the cases. That is a very big win for everyone. We now have two different tools that are very good at solving similar problems. Even if you can build a search backend in MySQL, you are probably going to have a better experience if you do it with Solr, for instance.
Disregarding well-proven technological solutions, with proliferous ecosystems, is dangerous. Failing to know new solutions that may prove to be better at solving your problem is just as dangerous. We really need to take the time to understand the problem that the new solution is fixing before jumping into it with both feet. Carey Flichel attributes this, amongst other causes, to boredom and lack of understanding in Why Developers Keep Making Bad Technology Choices.
It’s no surprise then that we want to try something new, even if we’ve adequately solved a problem before. We are natural puzzle solvers, and sometimes you just want to try a new puzzle.
Choosing the right solution often involves checking what everyone else is doing, and then analyzing the problem for yourself while taking all the options into consideration. You must not trust new solutions just because they're new anymore than you trust old solutions just because they're old. Instead, zero in on the problem to be solved and find the perfect solution regardless of the buzz. Keep every technology solution on the table until you understand the nuances of the problem space, and let that be your guiding light. Being aware of all the technologies involved, and knowing what’s the best choice, takes time and a lot of research. Many times this will require a software architect to guide you.