Great book on how big data could help build a better world. Although, I feel the author brushed over the ethical implications of snooping on peoples smartphones and internet usage.
It is not simply the brightest who have the best ideas; it is those who are best at harvesting ideas from others.Social physics posits social learning and social pressure as primary forces that drive the evolution of culture and govern much of the hyperconnected world.
Even though the use of digital networks has already converted the workings of our economy, business, government, and politics, we still don’t fully understand the fundamental essence of these new human-machine networks.
Suddenly our society has become a combination of humans and technology that has powers and weaknesses different from any we have ever lived in before.
The key insights obtained with social physics all have to do with the flow of ideas between people.
The spread and combination of new ideas is what drives behaviour change and innovation.
Social physics seeks to understand how the flow of ideas and information translates into changes in behaviour.
Examples include: financial decision making (including phenomena such as bubbles); “tipping point”–style cascades of behaviour change, recruiting millions of people to help in a search, to save energy, or to get out and vote; as well as social influence and its role in shaping political views, purchasing behaviour, and health choices.
Idea flow within social networks, and how it can be separated into exploration (finding new ideas/strategies) and engagement (getting everyone to coordinate their behaviour). Social learning, which is how new ideas become habits, and how learning can be accelerated and shaped by social pressure.
Rather than focusing on individual thoughts and emotions, social physics focuses on social learning as the major driver of habits and norms.
The scientific method as currently practiced in the social sciences is failing us and threatens to collapse in an era of big data
By better understanding ourselves, we can potentially build a world without war or financial crashes, in which infectious disease is quickly detected and stopped, in which energy, water, and other resources are no longer wasted, and in which governments are part of the solution rather than part of the problem.
Studies of primitive human groups reinforce the idea that social interactions are central to how humans harvest information and make decisions; ethnologists have found that almost all decisions affecting the group as a whole are made in social situations.
In humans, the social learning strategy of feeding back the best current idea—that is, a constrained, artificial sort of social interaction that interleaves periods of idea harvesting with periods when experts evaluate the ideas—produces a wisdom of the crowd effect that works even for small groups.
Some people clearly have impoverished opportunities for social learning because they have too few links to others. Others are embedded in a web of feedback loops, so that they hear only the same ideas over and over again, while most users have a middling number of opportunities for social learning.
As the pattern of connections between learners becomes optimal, the performance of the entire crowd improves dramatically. The result is a fractal dance of learning that spins ideas into wisdom.
A "star" performers’ networks are more diverse.
This is just what I see when I look at the most productive people in the world: They are continually engaging with others in order to harvest new ideas, and this exploratory behaviour creates better idea flow.
In a situation where people are uncertain, the confidence-enhancing effect of social learning becomes larger. This makes perfect sense: When people don’t know what is going on, they can learn by spending more time looking at what others are doing.
In situations when the outside information sources (e.g., magazines, TV, radio) become too similar, then groupthink becomes a real danger.
Exploration is the part of idea flow that brings new ideas into a work group or community.
Copying other people’s successes, when combined with individual learning, is dramatically better than individual learning alone.
A big danger of social learning is groupthink. How can you avoid groupthink and echo chambers? You have to compare what the social learning suggests with what isolated individuals (who have only external information sources) are doing.
One disturbing implication of these findings is that our hyperconnected world may be moving toward a state in which there is too much idea flow.
Idea flow is the spreading of ideas, whether by example or story, through a social network—be it a company, a family, or a city.
Each cohesive community has its own stream of idea flow that allows the members to incorporate innovations from other people within it, and even to create a separate culture.
Our behaviour can be predicted from our exposure to the example behaviours of other people.
Through social learning we develop a shared set of habits for how to act and respond in many different situations.
Are our habits the result of our personal choices or do they come from the flow of ideas that surrounds us?
Overheard comments and the observation of other people’s behaviour are effective drivers of idea flow.
As Nobel Laureate Daniel Kahneman might have put it, we can consciously reason about which flow of ideas we want to swim in, but then exposure to those ideas will work to shape our habits and beliefs subconsciously.
Moreover, in each case it appears that exposure to surrounding peer behaviours is the largest single factor driving idea flow.
Mathematical models of learning in complex environments suggest that the best strategy for learning is to spend 90 percent of our efforts on exploration, i.e., finding and copying others who appear to be doing well. The remaining percent should be spent on individual experimentation and thinking things through
Many commentators have observed that the power of social influence can lead people to both good and bad behaviours, and influence our behaviours to an extent that is scarcely believable.
Over time, the ideas of rationality and individualism changed the entire belief system of Western intellectual society, and today it is doing the same to the belief systems of other cultures.
Human behaviour is determined as much by social context as by rational thinking or individual desires.
Recently, economists have moved toward the idea of bounded rationality, which means that we have biases and cognitive limitations that prevent us from realising full rationality.
As a result, most of our public beliefs and habits are learned by observing the attitudes, actions, and outcomes of peers, rather than by logic or argument.
Working together also requires more than shared habits; it requires habits that result in cooperation.
How do we come to adopt habits of action that mesh together like pieces in a puzzle, allowing many people to work toward the same goal?
Synchronisation and uniformity of idea flow within a group is critical: When an overwhelming majority seem ready to adopt a new idea, this convinces even the skeptics to go along.
There is growing evidence that the power of engagement—direct, strong, positive interactions between people—is vital to promoting trustworthy, cooperative behaviour.
But social physics tells us that there is another way: by providing incentives aimed at people’s social networks rather than economic incentives or information packets that are aimed at changing the behaviour of individual people.
These incentives alter idea flow by creating social pressure, increasing the amount of interaction around specific, targeted ideas, and thus increasing the likelihood that people will incorporate those ideas into their behaviour.
The amount of direct interaction between two people predicts both the shared level of trust and the effectiveness of peer pressure.
We focus on changing the connections between people rather than focusing on getting people individually to change their behaviour.
The social pressure generated just by seeing what their buddies were doing doubled the effectiveness of the financial incentive.
Engagement—repeated cooperative interactions—builds trust and increases the value of a relationship, which lays the groundwork for the social pressure needed to establish cooperative behaviours.
Social physics tells us that we must include not only economic exchanges, but also exchanges of information, ideas, and the creation of social norms in order to more fully explain human behaviour.
If the majority of interactions were instead exploitative, then each interaction would serve to destroy trust.
Engagement requires: interaction, cooperation. building trust
Idea flow functions through social learning and social pressure to establish compatible norms of behaviour. And finally, social network incentives, which alter the dynamics of idea flow, can be used to efficiently shape the spread of new behaviours.
For almost all of the examples in this book, including the role of social influence on political views, purchasing behaviour, and health choices, as well as productivity in small groups, departments within companies, and entire cities, we find that using measures of the amount of social interaction—both direct and indirect—in order to estimate social influence produces accurate estimates of future behaviour.
Finally, a derivative of the model is currently used commercially to map the purchasing patterns of 100 million smartphone users (see http://www.sensenetworks.com, the Web site of a company I cofounded).
Idea flow takes into account all the elements of the influence model: network structure, social influence strength, and individual susceptibility to new ideas.
Group intelligence is about as important a factor in predicting group performance as IQ is in predicting individual performance.
The largest factor in predicting group intelligence was the equality of conversational turn taking; groups where a few people dominated the conversation were less collectively intelligent than those with a more equal distribution of conversational turn taking.
The second most important factor was the social intelligence of a group’s members, as measured by their ability to read each other’s social signals.
Women tend to do better at reading social signals.
The pattern of idea flow by itself was more important to group performance than all other factors and, in fact, was as important as all other factors taken together.
We found that three simple patterns accounted for approximately 50 percent of the variation in performance across groups and tasks. The characteristics typical of the highest-performing groups included:
This group problem-solving ability, which is greater than our individual abilities, emerges from the connections between the individuals.
Our study on the collective intelligence of groups revealed teams to be functioning as idea-processing machines in which the pattern of interactions facilitates a type of data mining of ideas.
Simply measuring a group’s interactions pattern allows us to accurately predict the eventual productivity of the group.
After instrumenting all these channels of communication, we then examined the interaction patterns within both high and low productivity groups.
What we found was that the patterns of face-to-face engagement and exploration within corporations were often the largest factors in both productivity and creative output.
We found that the most important factors for predicting productivity were the overall amount of interaction and the level of engagement (the extent to which everyone is in the loop). Together these two factors predicted almost one third of the variations in dollar productivity between groups.
Engagement is the central predictor of productivity. engagement — idea flow within a work group
Creative output depends strongly on two processes: idea discovery (exploration) and integration of those ideas into new behaviours (engagement).
Exploration is when team members interact with other teams.
Engagement is when they interact with each other.
The KEYS scale is widely recognised as the gold standard for measuring team creativity and innovation within organisational work environments.
I have found that the number of opportunities for social learning, usually through informal face-to-face interactions among peer employees, is often the largest single factor in company productivity.
In studies of more than two dozen organisations I have found that interaction patterns within them typically account for almost half of all the performance variation between high- and low-performing groups
We have found that engagement levels predict up to half of the variation in group productivity, independent of content, personality, or other factors.
Creative output is critically dependent on exploration.
Managers need to visualise the patterns of communication using dashboards and take steps to make sure that ideas flow within and between all of their work groups.
We found that group members with higher social intelligence enhanced the performance of the overall group across a wide range of tasks.
Effective leaders usually have a sort of practical charisma: By being energetic and systematically engaging with others, they can help grow the interaction patterns of their organization in the right direction.
"We found that the more of these charismatic connectors a given team had among its members, the better the team performance was judged during the business plan contest at the end of the week."
It seems that teams whose social style is dominated by these charismatic connectors may have had more evenhanded turn taking and high levels of engagement, which is the recipe for collective intelligence.
They tend to drive conversations, asking about what is happening in people’s lives, how their projects are doing, how they are addressing problems, etc. The consequence is that they develop a good sense of everything that is going on and become a source of social intelligence.
People can teach themselves to be charismatic connectors—they are made, not born. The trick is to do what creative people do: they pay attention to any new idea that comes along, and when something is interesting, they bounce it off other people and see what their thoughts are.
When we observe the fine-grain patterns of interaction within an organisation, we find mathematical regularities that allow us to reliably tailor the organization’s performance and predict how it will react to new circumstances.
The point is not just that it’s possible to get lots of people to work. Rather, the point is that it’s possible to get people to build an organisation that does the work.
Just as in our FunFit example, the behaviour of these editorial recruits was shaped by social network incentives, until they became a deeply engaged working group with standardized, shared ways of doing things.
Nevertheless, during the last century this sort of hierarchical crowdsourcing has been exactly the model of most corporations. Workers sit in cubicles doing independent tasks, and then their outputs are routed to anonymous others for the next stage of processing.
Because these types of organisational structures incorporate few or even no peer-to-peer network incentives, workers tend not to help each other learn best practices or maintain high levels of performance.
Changes in the network of interactions act like social network incentives, as the desire to reduce stress drives the development of new patterns of interaction.
Building strong ties with people is good for idea flow, but strong ties also can be used to exert social pressure.
When our language capabilities began to evolve, our existing signaling mechanisms were incorporated into the new design. As a consequence, our ancient social signals still shape our modern patterns of conversation.
We found that each of the different social roles that psychologists identify, i.e., protagonist, supporter, attacker, or neutral, uses different social signaling and, as a consequence, different patterns of speaking length, interruption of others, frequency of speaking, etc.
Not surprisingly, negotiations with lots of mimicry tend to be more successful, no matter which party starts copying the other’s gestures first.
Mimicry is believed to be related to cortical mirror neurons, parts of a distributed brain structure that seems to be unique to primates and is especially prominent in humans.
We have cities jammed with traffic, worldwide outbreaks of diseases that are seemingly unstoppable, and political institutions that are deadlocked and unable to act.
We need networked, self-regulating systems that are driven by the needs and preferences of the citizens instead of ones focused only on access and distribution.
Moreover, as smartphones continue to morph into personal information hubs that have greater computing capacity, they will reflect ever more information about human behaviour.
Together, wireless devices and networks constitute the eyes and ears of this evolving digital nervous system.
The proliferation of mobile phones makes it possible to leap beyond demographics to directly measure human behaviour.
Using data gathered from the digital bread crumbs that people leave behind, we can more readily answer questions such as: Where do people eat, work, and play? What routes do they travel? Who do they interact with?
But what are the overall patterns of our lives? The timing of our to-ing and fro-ing sets the rhythms of the city and determines peak demands on transportation, energy, entertainment, and sustenance.
The primary pattern is the workday. The second most pronounced pattern is the weekend. The third pattern, however, is a wild card: days spent exploring, usually a shopping trip or an outing.
Together these three patterns typically account for 90 percent or more of our behaviour.
We found that people’s behaviours change in regular, predictable ways when they are becoming ill, and that we can measure these behavioural changes using the sensors in mobile phones.
The ability to track diseases such as the flu at the level of individuals would give us real protection against pandemics, because we could take steps to reach infected people before they spread the disease further.
The way real-time flu tracking would work is by combining information from two sources: Data about changes in individuals’ behaviour patterns, because when people are becoming ill, we can measure predictable changes in these patterns. Location data, because physical interactions with others are the primary mechanisms for propagation of airborne contagious diseases.
Today we have a digital nervous system of sensors and communication already in place, ready to transform our cities into data-driven, dynamic, responsive organisms.14 Great leaps in health care, transportation, energy, and safety are all possible.
Urban centers are exceptional not only for depravity but also for innovation.
Cities are idea machines in the same way that companies are idea machines.
As we have seen in our work in companies, our closer social ties support engagement, because those people are more likely to talk to each other and provide the reinforcement that transforms ideas into behaviours. Likewise, our distant social ties serve the role of exploration, because we meet new people in new contexts and harvest new ideas from them.
The greatest chance of an experience that is new to everyone in their social network occurs at the places they visit least frequently.
It seems that not only does exploration result in growing more creative and richer cities, but the process is self-reinforcing. Greater exploration begets greater opportunities for exploration.
This suggests that when people have abundant resources, it is their curiosity and social motivations that drive their exploratory behaviour and not the desire to find cheaper prices or a better product.
The rate of idea flow is intrinsically a function of the ease of access and interaction between residents living in the same city.
In developing countries the average commuting distances are much less, suggesting that these countries could harvest huge gains in productivity and creative output from upgrading their transportation infrastructures.
But what if we could have both the high levels of social engagement characteristic of traditional villages (and hence their lower crime rate), and the high levels of exploration characteristic of sophisticated business and cultural areas (and hence their greater creative output)?
The goal is to get maximum exploration in the economic and cultural centers, along with maximum engagement in the towns.
We want to engineer the environment to enhance both exploration and engagement.
We have seen that today’s digital technology is not as good at spreading new ideas as are face-to-face interactions.
The reason behind this interest in digital media is that their low cost and scalability gives rise to the hope of a cheap way to manage companies, influence customers, and reach citizens. The answer is, of course, complicated. The key points to consider are trust and social learning.
Digital media don’t convey social signals as well as face-to-face interactions, making it harder for people to read each other, and so digital media are less useful in generating the trust needed for behaviour change.
When we need comfort or are especially happy, we want rich channels of interactions.
For instance, it is real-world interactions that drive most electronic interactions but once begun, electronic media can reinforce a trusted relationship, even though the people remain physically separated.
Personal data is the new oil of the Internet and the new currency of the digital world.
These data must not remain the exclusive domain of private companies, because then they are less likely to contribute to the common good.
We need to recognise personal data as a valuable asset of the individual that is given to companies and government in return for services.
Typically, no one is defined only by their job; people who come close to being that one-dimensional are considered odd, and perhaps slightly unhinged.
Anthropologists report that in the most remote and untouched societies they have found social traditions that are very egalitarian, with surprisingly equal sharing of food and often with distributed, expertise-dependent authority.
In other words, many early societies operated much more like an exchange network than a market: There was no market mechanism or price-setting authority for establishing the value of goods or ideas.
Exchange networks, buyers and sellers can more easily build up the trust that makes society resilient in times of great stress.
Social physics suggests that the first step is to focus on the flow of ideas rather than on the flow of wealth, since the flow of ideas is the source of both cultural norms and innovation.
I believe that there are three design criteria for our emerging hypernetworked societies: social efficiency, operational efficiency, and resilience.
We need to think more broadly, however, than simply how to rebuild damaged systems.
We also need to think about the resilience of the entire social design.
The poster children for such hidden dependencies are Lehman Brothers and AIG, whose collapse and near collapse demonstrated that most of the world’s financial systems depend on largely unnoticed and unregulated financial activities.
When decision making falls to those best situated to make the decision rather than those with the highest rank, the resulting organization is far more robust and resistant to disruption.
By creating social systems that are based on using big data to map detailed patterns of idea flow, we can predict how social dynamics will influence financial and government decision making, and potentially achieve great improvements in our economic and legal systems.
Imagine: We could predict and mitigate financial crashes, detect and prevent infectious disease, use our natural resources more wisely, and encourage creativity to flourish and ghettos to diminish.