Open source: it's not just for nerds anymore
When people say that software is open source, to most people it sounds like some esoteric tech term that doesn’t really make a difference to the average person, one way or the other. The reason that few outside of tech know how important it is, is because open source works so well, and the world is a much better place for it.
Let’s compare the software industry with some other ones, to get an idea of what happens without open source.
Perhaps the most stark contrast with software is biotech. We all have seen the amazing advances made in treating diseases made by researchers. Many diseases that were a death sentence can now be effectively treated with new drugs discovered by biotech scientists. But this can come at a very significant cost. Australia is shelling out $1 billion to bring down the cost of a treatment of Hepatitis C from about $100,000 to something affordable. In the US, many patients are denied coverage for this often fatal disease.
What is behind these prices? Of course it’s expensive to do research, but the prices charged are often not related at all to the costs. Witness the case of , who took a drug for babies with HIV suffering from a parasitic disease and increased its price by a whopping 5000%. This has nothing to do with intrinsic costs, but the ability for businesses to hold people’s lives hostage to their own financial ambitions.
Biotech is in a state where almost everything, even the most trivial biological advance is patented. Not only does this contribute to unbridled medical costs, but stifles innovation, because every step of a biotech process is proprietary. There has been an explosion in lawsuits in the biotech industry, with companies constantly suing and counter-suing each other.
Perhaps you don’t think that this is a fair comparison to what we see with software. After all, software isn’t as critical to society as medicine, or is it? Well, a lot of the explosive success of biotech is due to the use of computers for DNA sequencing, drug design and bioinformatics. And the vast majority of software tools that researchers use are open source and are critical to the marvelous discoveries that have been made. This is the part of biotech that is actually working!
But the importance of software goes well beyond that, being in every smartphone, laptop, router and car. And think of what it’d be like if somehow the open source movement never started. There were hefty fees for the use of operating systems, such as Windows, or text editing programs like Word. And there still are substantial fees today, but there are also a lot of free alternatives. You don’t have to be rich or work for a huge corporation to participate in this area, because an army of coders have spent decades writing all sorts of different kinds of software, often out of idealism, and not much else.
The reason that the open source movement started is not obvious. It was apparent to the few coders that started giving away their products, that they weren’t getting paid for their work. Purely out of idealism, they made sure that this was the case: These projects were specifically licensed to be completely free for anyone to use. The people originating this movement had a very clear vision for the future.
They started rebuilding completely free software systems from the ground up, so that they didn’t need to rely on any proprietary tools. And these were some of the hardest, most gnarly programs to write. Over the years, as the movement gained traction, more and more advanced open source projects were undertaken. The majority of contributors to these projects were doing this to free humans from the need to pay exorbitant fees to big corporations and allow anyone to be able to create and innovate freely.
Linus Torvalds, the originator of the Linux operating system said this to a revved up crowd in 1998. By then, 15 years after the start of the open source movement called the GNU project, Linux had become a viable operating system that had become a preferred programming environment for many software developers. Microsoft had started to pay attention to this fledgling open source product, and was not altogether happy. Linux turned out to be more efficient, less buggy and more secure than what Microsoft had to offer, but at the same time, Linux was completely free. At this point, businesses started seeing the general advantages of open source software.
When you make your source code available to people around the world, they can point out and fix problems, allowing it to evolve to a much higher quality product than if it had stayed proprietary. And in the 1990s, most operating systems being used were controlled by a near monopolistic corporation who had little incentive in improving quality. But with the rise of free software, this presented a significant threat to the bottom line of companies, particularly for the high-end server market where security and reliability are paramount. And as time progressed and servers became more numerous, big companies such as Google and Amazon chose to use Linux. By the late 90s, the writing was on the wall.
Today, Microsoft has embraced Linux, even providing a Linux environment inside the Windows operating system. How did this apparent coup happen? Because everyone realises that for many kinds of software projects, open source is a much better way to go. The long term benefits gained by sharing software and working on it collaboratively, greatly outweighs the short term profits that a company may gain by making software proprietary.
Many tech companies have now embraced the culture of open source. For example, Salsa uses Drupal to do its web development. Drupal is a very sophisticated and constantly evolving open source tool. There is a worldwide effort to continue to improve and develop Drupal which requires a significant investment of resources.
This is akin to building and maintaining a large public works project, for example a water reservoir. We all benefit from its presence and realise that we need to chip in and do our fair share. In the case of software, the amazing twist is that this massive collaborative effort is being carried out all over the private sector by tech companies around the world. This represents a seismic shift in attitude.
Tech companies still compete to build the best products and provide the best value, but we also recognise that there is a need to work together to maintain and build infrastructure for everyone to use, no matter if they’re in Stockholm or Mumbai. Despite the fiercely competitive nature of the tech business, we are actively collaborating on a myriad of open source projects for the benefit of everyone, and ultimately the consumer.
As stated above, in the early days, it wasn’t clear that there was any financial advantage to having free software. Naively one would expect that giving away millions of lines of code wouldn’t be a smart business decision. But sometimes idealism and the appeal of the Greater Good really does pan out. The world of open source has undoubtedly been instrumental in the extremely rapid pace of technical innovation. This is obvious to any tech person who has lived through these times, and now has an almost religious reverence for this collaborative view of technical innovation.
What we have seen is that the pioneering work and vision of a few people has saved us from the much bleaker future still seen today in other industries. This is a case where the good guys have won!
We live in a world where a significant fraction of software is open source and there are countless open source projects. GitHub, which hosts a lot of open source projects, has about 38 million different repositories. These projects range the full gamut of the human imagination, from video editors, to web browsers, to AI engines, to new programming languages, to e-business solutions.
One particularly important area of innovation has been in efficiency. Server farms, such as the ones used to house websites, consume of energy worldwide, and this usage is expected to continue to grow to even more alarming levels. To reduce this energy consumption, it’s important to make the software that servers use more efficient.
There have been a lot of efforts to innovate in that direction, and of course these are done largely using the open source model. For example the popular language Python, is used in many server applications, and a lot of development, such as “Cython”, gives a speed up of more than a .
Another example is the development of a relatively new open source language called Rust. It has many of the programming advantages of languages such as Python, but is often much faster. Benchmarks show it is .
Not only do these innovations use less energy, but they make server farms more economical, because they’re consuming less electricity and additionally, require less cooling.
It first might seem paradoxical, but the security used in computer systems is an open protocol. That means that the algorithms used to securely connect machines together are publicly available.
This has an enormous advantage over proprietary algorithms. One has to build a system assuming that the algorithm will be leaked, because if the security of the algorithm depends on all software developers keeping their work secret, then history has shown time and time again that the secret will become compromised. Instead, it is much better to devise a system that will work when the algorithms used are fully public. There are no global secret codes, just a secret identity for every user that is generated locally on separate devices.
And the beauty of open source is that the algorithms can be scrutinised by security experts around the world to see if they can find a flaw. Of course many have been found, and these are then patched by the open source community and updated. This has proven to be a very successful way of keeping the software battle-hardened, and much better than what was possible when proprietary protocols were used. At this point, open source standards have become ubiquitous and have transformed the way we can interact with each other, overwhelmingly for the better.
The ability to transmit information securely is done all the time now by billions of people. As you’re reading this, you’re using the https protocol which encrypts all of the information being transmitted to you, so that it's very hard for it to be intercepted by a third party. This has also allowed people to access sensitive information, for example financial transactions, without it being easily hacked.
This has also allowed the international payment of money to people often in developing nations, making trade much more efficient. Without open source security algorithms, we would have a few large corporations attempting to control hacking. The chances are that this would be much less effective than the open protocols that are used today.
Data encryption has allowed for people in many countries in the world to communicate freely with each other, without having to worry about Big Brother Watching You. The fact that the tools used are open protocols with long encryption keys, make it very hard for government entities, such as the United State's NSA, to outsmart them.
This secure communication has brought diverse groups of people together to openly discuss and freely exchange ideas. Dissidents in places like use apps built on these open source secure protocols to communicate, despite threats of extreme punishment if they’re found out. And of course, in more democratic countries, we have millions of social media pages where people communicate using secure network protocols.
During the pandemic, video communication kept isolated people in contact with their friends, and allowed them to work from home. This was vital economically and socially. The video and audio signals were also encrypted using open protocols.
Altogether the ability to communicate instantaneously with other people around the world using highly secure open protocols has had a transformative effect on people’s lives.
Open source formats and protocols
You might have heard of or remember the days of video tape format wars: VHS vs Betamax. Different companies competed for dominance in the video market by making players that could read either VHS or Betamax. This was an incredible waste in efficiency brought about by companies that were competing for dominance in this space.
Imagine what it would be like if you needed to buy a separate computer to access some websites, say ones run Apple software, as opposed to others running Microsoft. Why didn’t that happen? Because the engineers that developed the internet believed in open source standards. Researchers developed a small number of ways that everyone now uses, for transmitting information through routers. At the lowest level they developed a protocol to transmit information called packets. At a higher level the most commonly used method for ensuring reliable transmission of these packets is called . And finally you come to protocols that are used to access websites such as and the secure version of it discussed above, https.
All these methods of communication are done with open standards that have been agreed upon by virtually everyone. We don’t have to worry about buying a new device or expensive software to access certain bits of web content. Everyone realises how wasteful and silly that would be. Again, open source standards have won out, making electronic communications way more efficient than the state of affairs that happen when proprietary interests rule the roost.
The idea of open source software has inspired people in other areas to work towards the same idealistic goal. An example we’ll focus on now is how open source ideas have transformed the world of academic publications. This has had a profound change on science in general, speeding up the dissemination of discoveries, and allowing scientists to respond to critical situations, like Covid research.
In the 15th century the for the first time, allowed information to be disseminated in relatively large volumes from a single source. The advent of printing allowed for the distribution of scientific information, but of course, at a sizable cost. The way articles were published became a mainstay of academic life, and the situation remained pretty static for hundreds of years.
Publishing houses were making a decent living and didn’t seem particularly interested in fully utilising web technology to bring down costs to make research findings more accessible. To this day, researchers still pay hefty fees to see their research articles published. This is a problem for scientists without the economic means and is a barrier to global scientific communication. However the situation has gotten much better over the last couple of decades thanks to open source.
As the open source movement evolved a scientist, , saw the opportunity to make an electronic archive of scientists’ works. Through the award of relatively small grants, has been running for about three decades. At the moment, it houses about 2 million scholarly works that can be accessed freely by people all over the world. This has allowed scientists, including those in developing countries, to have their work read by everyone, without needing to pay publishing houses large fees. In comparison, the well known journal Nature charges authors more than $11,000 USD for open access to one of its articles. Even with 50% discounts for authors in developing countries, this is far from cheap.
The arXiv (pronounced "archive") doesn’t vet the content of archived work. It just allows you to post electronic preprints, making no claims about its validity. That still turns out to be extremely valuable because a lot of refereed and published work . Scientists always have to be skeptical of published papers, in any case.
As the popularity of arXiv.org grew, this created a lot of disruption for the publishers who didn’t react well to it. Publishers disallowed people submitting their unpublished work to the arXiv, because they didn’t want people being able to access articles for free. But then some journals decided this wasn’t very nice, and started allowing this practice. Pretty soon other publishers saw authors flocking to the nicer journals and realised that they better allow electronic preprints as well. After a few years, physics papers would be routinely posted to the arXiv before submission to journals. And some researchers even decided to only post to the arXiv, and forgo publishing in journals altogether.
The movement grew further, so that many publishers realised that they needed to make publications free to authors as well. Readers still need to subscribe to read the articles in these journals, but because the content was usually available on arXiv.org, the readers didn’t really need to pay anything either.
So arXiv.org managed to revolutionise the way that scientific publishing is done. It enabled researchers in many highly technical areas, such as physics, maths and computer science, to communicate their results quickly and for free and started a chain of events transforming the scientific publication industry.
Publishers in biological sciences and medicine reacted more slowly but change has finally come. Recently biomedical versions of the arXiv have become instrumental in communicating results about Covid. These are called and . Usually biomedical results take many months, if not longer, to get published in journals, and with the pressing need for new treatments and therapeutics, the old publication model just wasn’t cutting it. Now results from researchers all around the world can be shared instantaneously. Naturally, this has had a huge impact on the pace of discoveries in biology and medicine.
It is gratifying to see how the idea of open source in the publishing world snowballed, not only changing publishing but the way science is done, greatly increasing the pace of discoveries.
It's only the beginning
There are a lot of other great examples where companies have forgone short term proprietary gains to create more innovative, efficient and less wasteful solutions to problems. But still it is all too common to see businesses trying to make the consumer bound to them by clinging to the same old and wasteful business models.
If you need a part replaced for a mechanical device, say a car or a dishwasher, you need to buy exactly the same part number otherwise it won’t be compatible. And this isn’t because the slight differences between the shapes of these parts is important to the functioning of the machine. It’s because it’s in the company’s interest to make competitor or obsolete parts incompatible. Apple appears to routinely change their connectors leading to an enormous amount of waste. The EU has finally mandated that Apple . As we develop new technologies, such as electric cars, companies shouldn’t be using proprietary charging connectors. Charging stations for electric cars don't work for all brands. Does this make economic sense? Maybe for some car companies, but not anyone else.
It’s clear that there’s still a lot of opportunities for improvements in today’s world that can be enabled by the adoption of open source and related ideas. We’ve seen that open source has transformed the world of software and the more general cultural shift towards open collaboration, design and even , is making the world a better place. This shift in attitude leads to more rapid innovation, less wastage, higher efficiency and greater access to technology for people around the globe. Let’s hope that the spirit of open source continues to grow!