Central to my area of research are many things that are called complex. Complexity science and complex networks for example. For this reason I have become quite aware of the meaning of the word and therefor of its misuses. So I thought I try to explain what complex means in relation to simple and complicated and why it is so important to understand the difference in the modern world - I am really tempted to write complex modern world but it is kind of unfair to write about explaining a word and then use it without having explained it.
simple
So when are things simple? Sure enough one can use simple do describe many things. However there is a more or less good definition of what simple is in the context of complexity. Simple relationships between two things are those that one can easily understand, the ones that are obvious. One might say linear speaking in a more mathematical context. However I believe it is better to stay with the more vague definition of easy to understand and obvious. These might change from person to person but that only makes it clear that there is no clean border line between simple and complicated.
complicated
Complicated is in many ways the opposite of simple. This may sound impossible, since where is the place then for complex - but hear me out. Complicated things are those that we can not understand easily, that take time to think through. Some of the complicated problems are so hard that it might take years to understand them, but it is possible. Complicated problems are hard because they usually don‘t conform to our what we now about the world so far or because the chain of how one thing becomes another thing is much longer then we are used to.
examples
Now that we have both of these words covered we can start to make some examples. Lets have a look at a car. Using a car seems fairly simple, at least in concept. Doing it in the real world takes some time to get used to but everyone can do it. Back to the concept however.
In the simplest version you have two pedals and a steering wheel. You push one pedal to accelerate and one to decelerate, while you use the steering wheel to point the car in the right direction. Simple.
Even if you have a manual and have to operate a clutch and a stick it is still easy enough to understand.
If you look however how the acceleration part really works you will soon find out that it is not so simple. In modern cars it starts with a sensor that measure how far you pressed the pedal. This signal then gets processed by the ECU. Depending on the state of the car, it‘s speed, acceleration, steering angle, possible faults and diverse settings, the state of the environment like the temperature, oxygen in the air and legal requirements (hello VW scandal) the ECU then decides what to do. It does that by feeding all the data it gathers in to a software model of the physical engine it controls. This software model then computes how much fuel and which time to inject in the engine. And so on.
Depending in the model of car you can go down this rabbit hole at least a few more miles deep until you hit rock bottom. At several points there will be things that are not happening the way one would expect them to be assuming things are generally simple. However given enough time, we can understand all parts of this process and they will enable us to understand how the car accelerates if we press the gas pedal down.
When you ask yourself or others to make a list containing the big problems that mankind is facing, you will probably find these among them:
- Hunger
- Lack of clean water
- Climate change
- Availability of energy
- Disease
- Super-Virus
- Lack of new antibiotics
- Overpopulation
- Poverty
- Lack of education
- War
- Nuclear war
- Terrorism
- Stability
- Crashing economy
- Finanical crisis
- Natural disasters
- Man made disasters
Note that this is an unordered list, so the position has no specific meaning. Many of these problems are tied very closely together for example: hunger and lack of clean water are often due to poverty which is often caused by the lack of education, terrorism/war and or disasters. However the point of this post is to convince you that most of these problems are technically already solved we only lack the political willpower or solution to implement the solution.
Technically Solved
Lets have a look at the overpopulation and hunger complex. Both seem to be closely linked in peoples minds. We are to many, the world can not feed so many people. At the same time we waste almost as much food in the world as is consumed in Africa (222 million tonnes wasted vs. 230 million tonnes consumed). We do not have a food production issue we have a food distribution issue combined with a quality of production issue in some developing countries.
Considering that most of the food is produced and wasted in the developed countries however, mainly a distribution issue. This however means we have technically solved food production. The problem then must be poverty, if the people would have enough money they could buy the food.
So what about poverty?
Poverty is so apparent because of the distribution of wealth, the richest 1% of the people now own about 50% of the world. And the rest of the distribution is similarly unjust. As demonstrated by the hunger problem, it is not really the case that we do not have enough for everybody, we just do not know how to give it to everybody. Our attempts to do so have so far failed and ended in totalitarianism instead of communism. Please note, this is no attempt to make any claims that capitalism is inherently bad. I believe quite the opposite, that is, that capitalism has brought as very far – to where we are now – but we still need to evolve it.
Nevertheless, we have plenty of ideas of how to fix the problems we have currently in capitalism. There are proposals to change the current money and banking regime and change taxation in ways to keep the rich from owning 50% of the world.
Tackling these fundamentals of capitalism, money and property seems impossible (at least without pitchforks). Yet there are numerous example where it has been done before, our money regime changed last in 1971 due to decisions made by the U.S. government. After it was only introduced in the 1940.
An example for seemingly impossible taxation can be found in post war Germany where the actual wealth of a people was taxed. It was officially planed as means of to redistribute wealth. And we are not speaking about small amounts: up to 50% of a persons wealth could be redistributed spread over 30 years. Imagine that. One might think that this was only possible due to Germany‘s isolated position after the war. However the law was based on a Finnish law and Finland was not at all isolated after the war.
So again the conclusion could be that we technically know how to solve the problems of capitalism but we just can not implement them politically. Anyhow, wealth distribution is certainly not the only cause of poverty.
What about war and terrorism?
This is one of the things I really struggle with. Is there a technical solution to war? For many cases there actual might be one, however other conflicts seem impossible to resolve. Will the Palestinians stop to hate the Israelis once their economic situation is better? Would the Israelis allow the situation to become better? Will this end the terror? Impossible to answer.
We know that there are basically no wars in modern day and age that lead to an economically better situation, so one might assume that there are no wars in capitalism but this is obviously not true. Never the less, the number of people involved in wars is on an historic low (video), so maybe capitalism is also winning on this front.
Terrorism however seems to be another shoe. It seems to be the case the the western world constantly looses its battle against terrorism and the more wars we fight the more terrorists we create. And since the modern terrorism is basically founded as an answer against the wars the US and its allies are fighting in the middle east this should not surprise anyone.
In other words, terrorism is the evolutionary answer to war. We can not win against it with war. However there are plenty of ideas of how to fight against it, which again we do not implement because we can not bring it on politically. So lets look at some other fields.
(Note: I should link to something about this, however I don‘t feel like I have proper sources. A lot of my thinking however is based on ideas from this awesome podcast)
Diseases
There is no doubt that diseases are a problem – admittedly one I do not know much about. I know we have a problem with a shrinking number of effective antibiotics. I have seen how quickly a virus can get out of control (Ebola) and how infectious mutations (SARS/MERS) can be. However we have also seen how quickly we actually can develop a vaccination and discover even new classes of antibiotics or mechanisms of defeating harmful bacteria. So the real question might be why did we not manage to develop these things before a serious crisis was around the corner.
The answer is most of the time given in the form of capitalism: it is usually not effective for a company to develop a treatment that heals quickly. If you develop a new antibiotic drug it will be given to the patient for 1-2 weeks an that is it. A vaccine is even worse, a one time shot. If you develop a new pain medication, a new cancer or depression treatment you will likely make profit from the same patient for years. You can just not sell an antibiotic so expansive that it has the same return of investment. So where do you invest your money?
This makes the pharmacy industry look like bad people when in fact they are only doing the same as every other business. However we have plenty of ways to influence what the big industries are doing, it all comes down to the right incentives – as it always does in capitalism.
You want to keep your patents for your most grossing drug for two years longer? No problem develop something that helps against on of the things on this list we update yearly and you can!
Who can implement such a thing? Right, politicians. So what about the other big problem...
Climate Change / Availability Of Energy
This is a problem where we definitely do not have the technical solution yet right? Why in the world would everybody be working on a solutions otherwise? Wrong. I wrote at this at length in a previous post.
In review: you could harvest all the energy the world needs in a 500 km square in Africa. One would need to build thermo solar power plants, which would feed the power to methane production plants. The methane production needs CO2 and Water as input. The CO2 can be taken from the air and the water might be gathered through desalinization of ocean water. The compressed methane can then be shipped or pipelined in the world to be used like conventional natural gas.
Most of the infrastructure is already there, as are natural gas powered cars, buses, heating systems and power plants. The solar plants, desalination plants and even the methane plants already exist.
Just to make this clear, this is not a partial solution. If implemented this will be a CO2 neutral way of supplying the world with easy to store and transport energy until the sun burns out. This could put an end to climate change. At least to further damage.
Anyway, it is obviously not about to happen. Instead we have climate conference after climate conference no end insight. We also develop all kinds of fancy plans and complex infrastructure to solve it. So again, the problem is technically solved we just can not implement it politically.
But The Most Important Problem Is … !
I am more or less doing research in the energy sector. Therefore I usually argue like this: Availability of (cheap) energy is the most important problem we need to solve because it will solve all other problems. All we do is consuming energy, all things that make our life easier are consuming energy. We want everyone to have clean drinking water, no problem if we have cheap and "clean" energy we can just build a wastewater treatment plant or a desalination plant. We need more food? No problem we just build a greenhouse, temperature and light controlled and we are good. War? Mostly due to oil nowadays, once we need no oil anymore: gone. Disease? Once people do not need to care about oil and war they have more time to work on disease prevention … and so on. And you will find a lot of other very clever people arguing the same way for energy.
Or for war, or poverty, or hunger …
In fact all of these problems are somehow connected, you can not solve one without the other. This does not mean that there is one root problem connecting them, they are all real problems on their own. However the connection between most of them is politics …
So Should We All Become Politicians Then?
Definitely not. While there are some former scientist in politics, some even from natural sciences, I believe not all of us are suited for a job in politics. However most researchers should radically change the view on how to do research. There is one part of science which is tasked with finding out new things about nature, sometimes very fundamental things like the particles that make up the universe, sometimes more about tiny details like how two very specific molecules interact. The another part of science however is tasked with finding solutions, often to the problems discussed here or related ones, or even completely unrelated ones. There might not always be a clear distinction between these roles and the same researcher or group may switch between both of them rapidly. However most of the time it is very clear if the outcome is a solution to a problem or a new fact about "how things work".
Now every time you develop a solution to something you have to consider the politics that might be needed to implement your technical solution – because if your solution is political not possible to implement it is just not a solution. For other fields this kind of thinking has become very common, every time a new engineering solution is developed someone will check if it is cost-efficient. From the engineering standpoint this is not necessary if it technically works, the problem is solved right? Except it isn‘t because if it is not a profitable solution for a company it will not be adopted. The same is true if the solution is not feasible in a socio-political sense.
Therefore in the same way engineering and many other disciplines have adopted economic theory we need to adopt socio-political thinking to test if our solutions are actual solutions to a problem not just technical ones.
In many ways this means to just adopt more from economic theory than just cost calculations, because at the core economics deals with the incentives people have for making their decisions, it just happens to be the case that incentives often come in the form of money. Nevertheless incentives and rules are what politicians can use to make a difference, however very few technical solutions we have come with a bundle of incentives and rules that need to be in place in order for the solution to work right.
Examples?
To give a more concrete example, we just saw the introduction of distributed energy storage for the mass market. However the battery technology is not yet cheap enough or good enough to go off grid. But since we have a lot of solar and wind energy in certain countries, resulting in high price fluctuations for power, these batteries start to make economic sense in a way that they can be cost effective. Which is why we will most probably see them adopted in many places.
Once we have enough batteries they will however begin to influence the price of electricity, because they dampen the price fluctuation we currently see, by buying power when it is cheap and selling it when it is expensive. Which means once we have enough batteries the fluctuations will reduce considerably if not vanish, which is very beneficial for society because it means energy is always available in enough magnitude. We have to remember however that these very fluctuations are what made the batteries worth buying in the first place. Economic theory now suggest that people will stop buying batteries because it is no longer profitable.
The problem however is that the cost effectiveness of the batteries was calculated over many years, so once at the point that the price fluctuations begin to shrink, it is already to late for all the buyers of batteries. Economic theory again suggest that we all try to optimize our profit, how could you do that if you have bought a battery? Fairly easy, you just try to sell as late as possible when the price starts to rise, because as long as no one is selling the price will continue to rise. So there is now an incentive and a possibility in place for all the battery owners to increase the fluctuations instead of decreasing them. There are several reasons why you can not play this waiting game indefinitely, first of all at some point the grid will have a blackout. Second, the regulated power plants will kick in and supply the power needed. Third, if enough of the other players sell before you, you will not make your needed profit. While the third reason might keep the whole situation in check, by keeping everyone on its toes while waiting, it does by no means guarantee stability. Because now everyone is looking for a change in the speed of the rising price, because a slower rising price suggest the others have begun to sell. So now a slowdown might trigger an avalanche of sales, flooding the market.
So instead what we need is a regulations that makes it very unlikely or downright impossible for the owners of batteries to engage in this kind of speculation, while still giving them an incentive to buy a battery helping to stabilize the power grid. It is exactly this kind of political solution that is needed for the technologies to succeed and be of value for all of society. We engineers are problem solvers, however we leave some of the toughest problems to figure out for the politicians – even though we know they fail to solve them a lot of the times. This has to change.
My current work involves some agent based simulation written in Python. I now wanted teach it some new tricks including running on multiple cores. While my normal approch for multithreading in Python is to just run several simulations in parallel to get some average, I decided it would now make sense to also have single simulations run on multiple cores. One reason for that was to make it scale more easily on the computing cluster, the other was that I assumed it would be quite easy with the agent based stucture.
For the same reasons I decided to use MPI (mpi4py): it seamed like a good fit for an agent based model (agents sending messages around) and it was available on the computing cluster. It turned out that in the end I did not use any mpi code in the agents to keep them more flexible.
Anyway, the simulations is structured in the following way: on each core/process a controller is initalized which handels not only the communication but is also in charge of the agents. This is a good fit because the simulations involves global state as base of the agents decisions.
I decide in the beginning which controller controlls which agents. This might lead to some unused processing time during the sync between the controllers because they might have to wait to get the globals state. But since each agent has a lot of data attatched, I assumed it would not make sense to depatch the agents to a different process every timestep. This would mean to shuffle around big python objects and therefore presumeably slow.
Instead only the global state is shared between the controllers. I am not sure if this is the best solutions since I did not try any other but it turned out to have a nice structure and work quite well.
One problem I sumbled upon was the that I could not gurante that each group of agents would be the same size, because when you want to simulate 100 agents on 16 cores it does not add up. This meant I could not use the normal allgather methods of MPI to collect the state but had to use allgatherv which can deal with the different sizes. Sadly there is no documentation about allgatherv in MPI4PY. With some googling I worked out an example of how it works I wanted to share so here it is:
from mpi4py import MPI
import numpy as np
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
mpi_size = comm.Get_size()
# work_size = 127 # arbitrary prime number
work_size = 100
work = np.zeros(work_size)
base = work_size / mpi_size
leftover = work_size%mpi_size
sizes = np.ones(mpi_size)*base
sizes[:leftover]+=1
offsets = np.zeros(mpi_size)
offsets[1:]=np.cumsum(sizes)[:-1]
start = offsets[rank]
local_size = sizes[rank]
work_local = np.arange(start,start+local_size,dtype=np.float64)
print "local work: {} in rank {}".format(work_local,rank)
comm.Allgatherv(work_local,[work,sizes,offsets,MPI.DOUBLE])
summe = np.empty(1,dtype=np.float64)
comm.Allreduce(np.sum(work_local),summe,op=MPI.SUM)
print "work {} vs {} in rank {}".format(np.sum(work),summe,rank)