Variety and organization (long post is...)

[davv]

Elsewhere, I referred to how useful I've found variety in explaining why organizations behave the way they do. In particular, I thought a bit about why nations with large states tend to encourage the people to be uniform, through a sort of "equality by uniformity". And then I thought about variety, and just by turning the problem around, looking at it from the right angle, it became simple... and it should show what it is I mean.


To make sense, I'll have to show what I mean by "variety". Variety is the number of possible states of a system. A ball resting on a table has only one state, where it is resting on the table. A heater may have more than one state: it can be on, off, or somewhere in between. If the only choices are full blast or nothing at all, then it has a variety of 2.

I also have to show what is meant by control, in the technical sense. A controller or regulator limits the variety of the system it is controlling. A room without a thermostat-controlled heater has a wide range of possible temperatures (depending on the temperature outside), but one where the temperature is being regulated has a much more narrow range.
The controller does this by observing and adjusting towards the desired outcome, then observing again, adjusting further, and so on. This is feedback.

So, how does that relate to organizations and governments? There's one more thing to know: in order to control a system, the controller must have at least as much variety as the system[1]. For every move that may take the system out of the desired state, the controller has to have some way of pushing the system back into the desired state. If the controller does not, it will only partially control the system, or fail entirely. That is requisite variety: "variety absorbs variety", or "only variety in R can force down variety in D".

---

If we take the traditional ideologies of left and right at their face value, then it appears that the distinction, when taken to the extreme is as follows:

• The right ideology thinks that society should be like a cold-blooded animal: to have no regulator, but instead simply go with the dynamics of the system itself. Therefore, it's better to have a free society than to control it, even if the latter could give a better foundation.


• The left ideology thinks that society should be "warm-blooded" because the evolution of society, left to its own premises, will result in very bad outcomes (extreme wealth inequality followed by revolt and collapse). Therefore it's better to have a reasonable foundation to keep the system from going out of spec, and if that requires a regulator, so be it

The "regulator" in this case is usually the state. However, the state is organized in a bureaucratic manner, which means it has low variety. Unless the state is a police state, it is limited in what actions it can take, and even for those actions it can take, the inefficiency of the bureaucracy means that the "countermove" is only partially successful in limiting the variety of, say, the economy.

Since the state has low variety, the ideologies above are polar. If the regulator is to permit a high variety in society, it must accept a weakening of its ability to regulate or control. On the other hand, if the regulator is to be successful at what it aims to do, which is to limit unfortunate outcomes, it must somehow manage to have society lower its own variety because the state cannot directly handle its task otherwise - it simply does not have the capacity to do so.

Out of that observation, the tendency of left-leaning central governments to desire uniformity becomes easy to see. The state cannot directly control the parameters it wants (economic ones, for instance), and so will promote what I call a type-2/type-3 or majoritarian concept of equality: that everybody is equal if they're uniform. If the people accepts it, that would greatly reduce the variety of society, potentially bringing control within the grasp of the state[2].

One would also assume, based on the above, that right-leaning governments would forgo regulation altogether. That is certainly a goal of its extreme position, but "right" seldom goes as far as to advocate a free-for-all anarchy, for that would destroy freedom. Even the right wants some regulation, and practically speaking, the economic powers (corporations and private organizations) that tend to benefit from a right position will advocate that they, rather than the state, perform the task of a controller, but for their own purposes.

---

Even though I didn't specifically intend to reach the conclusion when I first wrote this (in a text file of my own), it's surprising how easily the conclusion follows: if the state doesn't have enough variety to regulate well - which it doesn't, being bureaucratic - it will encourage the system to become more well-behaved so it can.

As for me, I think that if left to its own, the system, at least the economic part of it, is unstable. If there is minimal economic regulation, some entity (a company, most likely) will eventually gain enough power to rewrite the rules. At that point, it can decide to be the new state, and the control it then exercises will not be for the good of the people. Pretty much the same thing applies in the absence of any order at all, except the entities are not only economic, they make use of physical force as well. The different mafia gangs will compete until someone ends up on top, at which point the game is over.

The alternative, though, is not very good either. It is better than being in a cyberpunk future, but if the state cannot regulate directly, it will keep trying to simplify the people so that they can be regulated. What is the answer? I think, and this is my opinion, of course, that the regulator must be set up so that it has the requisite variety - not by making the state a police state, but by design of the organization, decentralization, and so on. A recursive design would help, but it must not become too bogged down in sending messages up and down a hierarchy. With the power granted by the high variety in the regulator, the regulator itself (the "state", or more broadly, the group of organizations handling governing functions) has to be anchored by the wishes of the people. Ideally, it should be radically transparent, free of hierarchy, and incorporate ordinary people so that it will resist corruption for as long as possible.

---

[1] Generally, it must have as much variety as it's trying to reduce in the system being regulated. A bang-bang (on-off) thermostat can limit temperature in a room to a certain range by using some variety, but a PID controller can make use of the heater's ability to be set at other levels than full blast and nothing at all to regulate to a much tighter range.

[2] The more centralized the state, the more variety it has to deal with, and the greater this effect would be. The effect is the same thing that makes micromanagement so impractical. If a center tries to coordinate everything, even if that center is somehow perfectly honest and not corrupted by power, it soon finds itself overextended and quality suffers.

Comments

[draque]

There are two points that you make that seem like they don't reconcile with one another. You acknowledge that with a larger centralized state, high variety has to be dealt with (which becomes difficult, as choices become increasingly expensive to add as a system grows). Hierarchy is the current answer to this problem, as it isolates variety by region and governmental layer, but the second point I wanted to address was where you say your system would be as non-hierarchical as possible. I feel like this isn't something that you've overlooked, but it's not something that I understand. How would your system handle that?

[davv]

Hierarchy works through a separation of concerns, where the top layer give general specifications and the lower layers deal with what is their concern. A replacement system would also use separation of concerns, but would permit feedback in multiple directions.

A state might have a large body that makes general decisions and appoints smaller bodies to implement those decisions, recalling the bodies if they do badly. This sounds like a hierarchy, but the "top" (where power resides) is made up of those who are affected by the decisions of the implementers. A parliamentary democracy is somewhat like this, because the executive is appointed by the legislature which is elected by the people, the legislature makes more general decisions than the executive, and the executive can be replaced upon a vote of no confidence. However, in a parliamentary democracy, the people can't recall members of parliament, and the feedback upon the parliament by the people is too weak (on the order of a few bits every election period).

Another way of separating concerns would be to do so geographically. The local government elects/appoints that which manages local concerns. Invariably some decisions will not be purely local, so these escape the system and are handled by a regional group, held accountable to the people of the region as a whole. Some decisions escape *that* system and are handled by a national group, and so on. To draw a similarity to the body: we usually don't notice what's going on with autonomous functions unless something needs our attention (because the conscious mind would be inundated otherwise); and similarly, the national group or people as a whole wouldn't have to deal with all that variety unless the regional ones can't manage.

Yet another would be by specialization. A broad organization could set objectives and a playing field and then admit more specialized organizations - sort of like a regulated market. In this approach, there is a trade-off by redundancy. If there are many organizations dealing with the same thing, they can't make use of the economics of scale, but if there are few, they may escape incentives based on competition.

These structures are not foolproof. If the regulator has too low variety, they will fail in some way or other. The implementers may hide specialist specific knowledge to steer the people in a certain direction, the national government may argue in favor of efficiency, market participants may redefine the rules, and so on. But neither is hierarchy. A bureaucratic hierarchy is quite bad at adapting to the world, because it is filled with many minor bosses who want to keep their power; and a brute force hierarchy, while very good at translating specific directives into action, can lose contact with the world as information travels up the pyramid.

Thus, no matter what model (or combination of models) is to be used, I also think we'll have to increase the variety of the organizations/state, so that these kinds of errors don't appear, or appear less often. To do so, one could use automation, either to replace or to augment the bureaucracy; but the computers would have to be used in the proper way to make the picture simpler rather than more complex.

I say that the system "has" to be anchored in the people. There's no law of cybernetics directly saying this; applying cybernetics to organizations and states just suggests how one is to make a regulator that takes some specifications and holds a system within them, and what is required for the state to do so. However, if the specifications were set by a Dear Leader instead of the people, especially if the state is of much higher variety, the dictator would become mad with power and "introduce instability". In other words, he would make such strange decisions that the people would not put up with it and there would be a revolution -- that, or the police would quash it and the nation would suffer. If the dictator disregards that and considers his own well being the only metric, then that's irrelevant, of course, so it's not an absolute, but it's pretty close to one.

[draque]

Ah, I can see what you're getting at there. It seemed at first that you were saying that all levels of hierarchy should be removed. The only element left that I would be a bit wary of is the computer based elements of governance. No programmer or designer can ever be relied upon to be more clever than every subsequent human looking for a way to game a system. Rather than making it a computer based system (which will inevitably involve methods that are unseen by the majority of people/exploitable by the minority), couldn't the same results come from a set of legislative rules regarding distribution of authority? You point out that the purpose of the computer system is to remove or replace bureaucracy (which, admittedly, the legislated rules would increase), but it seems a weak point to me, and the first place that I would search for methods of attack if I was a politically motivated person. And politically motivated people are one of the constants of humanity, unfortunately ^^().

[davv]

It is true that those who'd want to benefit from the system might find a way to game it; but that is true of the bureaucratic system, as well. The comparison of an organization to an AI with human base components is more useful than one would think of at first, and in a bureaucratic system, one can also exploit the components through attacks like corruption, bribery, "networking", and so on.

So, if a bureaucracy is supposed to be impartial and carry out its tasks no matter the inputs, it's similar to a program; and the less discretion the bureaucrats can exercise, the closer the similarity. Like you say, programs can be exploited, and these "programs" can be, too.

The difference between the two would be that automation would more rigidly separate the tasks of specifying and implementing, and when they could deal with higher variety, would also be more powerful and so should have even stronger safeguards.

The decisions and actions of the government should be public unless there's a reason not to (usually involving privacy), not the other way around. This would include the data used to calculate if that is done. The programs' source should be publicly available, and for specific calculations, it should be possible for independents to verify the result by using the data (which is made public) and the program (which also is). The actual mechanism of balance is that the public (or the parts of the public that would understand the programs) would be more interested in verifying the results if the decision is controversial or the question is important, and that reasoning should work unless there's a massive conspiracy of programmers and media.

In some cases, it would be possible to invisibly bias the system. If the system of elections to a legislature was replaced with one where everybody can directly vote on bills or pass their power to somebody else (and move the power elsewhere at any time), then someone might hack the system to give undue power to certain people by a sort of salami slicing attack. (I'm not necessarily saying this is a good system, but it's one that would be impossible without computers, and so shows the point.)
If that is the case, one is left with "ordinary" investigation. The hackers would have to plan and probably contact the person in question, which would leave traces; or sysadmins might have noticed attacks on their computers, and so on. The fact that power was being siphoned would not be discovered until later, but that is usually the case with attacks and crimes in general today, too.

For some kinds of software, there would be little point in hacking it. Hacking a web forum or wiki that people use to coordinate initiatives would be annoying in that it would deny service, but it would easily be discovered.

-

So to sum all that up ... :)

An automated bureaucracy would be more powerful than a manual one, but they are similar to the extent bureaucrats can't exercise discretion. To the extent they can, people would still be able to specify after the change, and to the extent the bureaucracy is rigid, the rules that keep the bureaucracy balanced could be adapted to apply to the use of the programs as well. With transparency and freely available source, the integrity of the actual calculation could be independently verified if required.

[draque]

Ahh, I can see what you're saying there. I suppose I do more or less agree with it overall, but I'm still wary of computer automated systems like that, simply because they're unapproachable/not understandable to the general populace. That having been said... I can also see that complex systems of government are just as unapproachable. Ultimately, I suppose that the computer system in this particular case could have a rigidity that would be incorruptible... but I would worry that the same rigidity would make it vulnerable to system-gamers, once they found exploits that hadn't been considered when it was first put together.

[davv]

That's the same question as lies in whether or not to disclose security bugs to everybody, only to the producers, or not at all. Different situations have different answers (I like how this paper shows when the different approaches work), but I think the transparency would make it at least as easy for the "many eyes" to discover problems as for those who would exploit the bugs in the incentive system.

Or at least I hope so :)

(Even if not, it may still be worth doing the trade to get the agility that automation can provide - while one can run programs on any computer, one can't make a copy of all the bureaucrats of a department and experiment with changes in laws.)

[lhexa]

There's one more thing to know: in order to control a system, the controller must have at least as much variety as the system[1].

I think this is wrong, but could be reformulated as: a controller must have at least as much variety as the end state it seeks to bring about. To go on, you said:

For every move that may take the system out of the desired state, the controller has to have some way of pushing the system back into the desired state.

This doesn't support your point, because the same way of "pushing the system back" may be used against different ways of moving the system out of the state. To use a hopefully realistic example, a patriarch who sets up an arranged marriage for his daughter does so in part by preventing her attempts at other marriages: one method of control works despite the variety of the people she could otherwise marry. Or, a society may attempt to discourage theft by cutting off the hands of thieves, but theft is a much more varied activity than this punishment. Here I'm hampered by the fact that there are no reliable ways of controlling human beings, though, so the topic might be about something that doesn't exist.

Regarding your last paragraph, I would argue that we should be seeking out means of social shaping that are not matters of control. One minor insight which could be used in this goal is the fact that no human activity ever disappears entirely from human life: so if you want to eliminate the worst of that activity, do not suppress it but instead make it figurative or abstract. If human beings are given ideological wars to wage, they might not get involved in actual wars anymore; if the human impulse to control things, to set up rules and impose order, can take place on the scale of small (say online) communities, perhaps that will thwart the tendency to impose control on a larger scale. And perhaps if people can be violent in games, they'll be violent in life less often. I'm inclined to offer this as an explanation for the steady decrease in violent crime in the US since around 1990.

This comes back to our old discussions of anarchy... I think that advances in human society will come about as people discover ways to achieve social ends without imposing control, in the long run hopefully leading to a government based less on power.

[davv]

I think this is wrong, but could be reformulated as: a controller must have at least as much variety as the end state it seeks to bring about.

That doesn't really parse, because the variety here is the number of possible states. A single state doesn't have variety; it is the system which has variety. The regulator responds to a perturbation by picking the appropriate action to cancel that perturbation out, so that the regulated system stays within parameters.

A more proper statement might be "the controller or regulator needs to have at least as many state transitions as the system being controlled in order to unconditionally control that system". If you're content with controlling the system most of the time, variety becomes probabilistic and might look quite a bit like information entropy. The controller tries to correct to a state by compensating for perturbations like an error-correcting code tries to correct to a known message by compensating for noise, and if the bandwidth isn't up to it, the objective is impossible.

This doesn't support your point, because the same way of "pushing the system back" may be used against different ways of moving the system out of the state.

I see your point, and I can think of a more mechanistic example, too. Say you want to keep a room lit and not too hot. The controller turns on the lights when it gets dark and a fan when it gets hot. Now the variety is roughly speaking balanced - there are four states (two bits) of interest in the system and also four states in the controller (i.e. what's switched on). However, you could do the same, albeit ineffectively, by making the controller turn on the fan and the light if it either gets dark or hot, since the presence of light doesn't interfere with temperature, nor does the fan make it not-cool if it's already cool. Thus the controller only needs to distinguish between two states (hot or dark or both, everything else), and only needs two states itself.

(I'm being a bit quick here since such a controller would be inconvenient since it would cool the room too much if it's already cool, and it might trip itself up thinking it's no longer dark when it turns on the light, but you see my point, hopefully :) )

What's left unaccounted is the relative power of each action: the "buffering" or degree to which the actions or circumstances themselves regulate the situation without feedback. Again, I can think of a mechanical example: if you insulate a room, that "regulates" the temperature without feedback - the insulator needs no variety at all! Or if you have a controller that gathers rain when it's raining, it doesn't need to know exactly how much rain falls because the short term variety is averaged out in the container where it gathers that rain.

I wonder how that would translate into the error-correcting code concept? It's not altogether intuitive.

Regarding your last paragraph, I would argue that we should be seeking out means of social shaping that are not matters of control.

Ideally, we'd want a situation where no shaping at all is required. If we could make the problem of allocation irrelevant by having an economy of abundance, that would (all other things equal) be preferred to any attempt to solve the problem of allocation itself (through price rationing, etc). If I were to generalize that reasoning, making the problem of regulation go away would be preferable to finding the best way of doing that regulation, so your point would follow, yes. If both of these fail or are inapplicable, though, I think the kind of reasoning I wrote in my post would help constructing a better controller/regulator, avoiding the usual traps of corruption, excessive centralization, lack of accountability, and so on.
At least, unless we find ourselves in a utopia, the higher level maintenance - deciding where to go, how to shape society, then having that change happen - involves departing from the natural equilibrium, and thus some sort of steering will be required. That steering or control doesn't have to be as heavyhanded as a state backed up by force, but it would involve feedback.

So yes, in the long run, may we find ourselves not needing to actively adjust the trajectory of society to keep it within what's desired; but as long as that has to be done, it would be really nice if the organizations could work better. As long as there is a bureaucracy, may it have the requisite internal variety so that it doesn't try to simplify the world and force everybody into a few preset molds, or one, to make its own work possible.

[lhexa]

That doesn't really parse, because the variety here is the number of possible states.

It was poorly worded, but your A/C case was a clear counterexample anyway, so I'll withdraw the point. The controller can have greater or lesser variety than its system.

Ideally, we'd want a situation where no shaping at all is required.

Probably you mean something different than I do by "shaping", but that statement just sounds implausible. For instance, language shapes society, and even in a utopian one you'd want to continue improving language, say by incorporating new terms from developing sciences into common parlance. If you meant deliberate shaping, I'll note that the hypothesis I gave (that video games helped decrease violent crime in the US) wasn't a deliberate effect.

If both of these fail or are inapplicable, though, I think the kind of reasoning I wrote in my post would help constructing a better controller/regulator, avoiding the usual traps of corruption, excessive centralization, lack of accountability, and so on.

*nods* Yeah, I think so.

At least, unless we find ourselves in a utopia, the higher level maintenance - deciding where to go, how to shape society, then having that change happen - involves departing from the natural equilibrium, and thus some sort of steering will be required.

I question this. Societies generally improve and develop without steering -- the progress of science is a great example. Societies also push themselves out of equilibrium without any kind of steering, sometimes for the better, sometimes for the worse.

[davv]

Probably you mean something different than I do by "shaping", but that statement just sounds implausible.

To be more precise, by shaping I meant using something that requires continuous maintenance in order to pull the system from its usual equilibrium. Ideally, we'd have (or want to have) a situation where the equilibrium takes care of itself.

This gets a bit more muddy in the case of a society because it is "shaping" (in your sense) itself all the time, which I suppose is your point. Yet from a cybernetic point of view, you still have controllers - it's just the case that these are less separate from the system than you'd usually consider the case for say, a state, because the controllers are the members of society acting upon each other (individually or collectively).

So, one step removed from ideal, you have the "establish the environment and leave" sort of regulation, where you'd have a minimal code (either social or legal) which then results in what was wanted. For instance, the diverse dynamics of capitalism arise from seemingly simple rules[1], and one might imagine that say, adding a rule (whether social or legal) that organizations should (or have to) disclose input-output data, would move the economy closer to a planned form -- without having to build planning agencies before the fact, or impose production orders upon the producers themselves.

I question this. Societies generally improve and develop without steering -- the progress of science is a great example. Societies also push themselves out of equilibrium without any kind of steering, sometimes for the better, sometimes for the worse.

Hm, I'm not sure. It seems to me that as you increase complexity, there are more and more ways things could go "wrong" and fewer ways they could be right. The paths multiply, and it generally seems the buffering technology (that which would work like insulation of the wall) lags. On the other hand, evolution manages to increase complexity on its own, and that is a simple process. I suppose if you leave a system to itself, it will attain some form of equilibrium (and if it degrades from that point, well, that wasn't the equilibrium after all). Whether it's better than what you could have by adjusting it beforehand, taking the possibility of corruption of the adjusting mechanism into account... is another question.

I'll have to think about it further, it just again seems intuitive that one could make shortcuts by coordination, and by knowing how controllers work, one could put the right amplifiers in the right places and get much less corruption/etc; and then, if society becomes reliant on those shortcuts, that means it needs the mechanism just to stay at its current level, unless it later develops a passive ("buffering") solution.

-

[1] and possibly a state to enforce them, that is. Absent a state, the producers would turn on the rules pretty quickly, which one might consider a part of the greater dynamic - but if one does, then that greater dynamic is unstable.

[lhexa]

I'm going to back out of this thread, my apologies.