Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

Cambridge University Press eBooks

Introduction

Massimo Franceschetti

Published Dec 4, 2017• 0 citations
Publisher

Abstract

Nihil est in intellectu quod non prius fuerit in sensu. The Physics of Information This book describes the limits for the communication of information with waves. How many ideas can we communicate by writing on a sheet of paper? How well can we hear a concert? How many details can we distinguish in an image? How much data can we get from our internet connection? These are all questions related to the transport of information by waves. Our sensing ability to capture the differences between distinct waveforms dictates the limits of the amount of information that is delivered by a propagating wave. The problem of quantifying this precisely requires a mathematical description and a physical understanding of both the propagation and the communication processes. We focus on the propagation of electromagnetic waves as described by Maxwell's theory of electromagnetism, and on communication as described by Shannon's theory of information. Although our treatment is mostly based on classical field theory, we also consider limiting regimes where the classical theory must give way to discrete quantum formulations. The old question of whether information is physics or mathematics resounds here. Information is certainly described mathematically, but we argue that it also has a definite physical structure. The central theme of this book is that Shannon's information-theoretic limits are natural. They are revealed by observing physical quantities at certain asymptotic scales where finite dimensionality emerges and observational uncertainties are averaged out. These limits are also rigorous, and obey the mathematical rules that govern the model of reality on which the physical theories are based. Shannon's Laws Originally introduced by Claude Elwood Shannon in 1948, and continuing up to its latest developments in multi-user communication networks, information theory describes in the language of mathematics the limits for the communication of information. These limits are operational, of real engineering significance, and independent of the semantic aspects associated with the communication process.

DOI
10.1017/9781139136334.003

Related papers

The Physics of Information

Torsten A. Enßlin, Jens Jasche, John Skilling

What is information? A short answer could be: Anything that changes our minds and states of thinking! Information is virtual. It can be carried by speech, an image, scratches on a stone, patterns in a photon field, the connections of neurons in our brains or even by the wavefunction of an electron. Information does not depend on the actual means of transmission nor does its identity change when changing the carrier. Information is physical. It needs to be sustained by a physical substrate and rearranging that substrate to contain the wanted information requires to spend energy or work. Without a physical world, information can not be stored, processed, or transmitted. Our world is permeated with different representations of information. Actually Information about the physical laws governing our Universe can be found everywhere and in everything. Could it then be that the real fundamental elements of this world are tiny bits of information? Could we get it from bit?1 A number of researcher investigate this possibility, and several of their attempts to recover physical laws from the principles governing information are reported about in this special issue on physics of information. The articles by Gregg Jaeger,2 by Ariel Caticha,3 and by John Skilling and Kevin Knuth4 address the reconstruction of quantum mechanics from information theory. Furthermore, Kevin Knuth and James Walsh5 show how to recover space-time from particles sending each others signals in an initially geometry-free setting. A central element of most works on information is the concept of a probability. Probabilities can be defined via various routes. Most familiar might be their definitions by the frequentists school as relative frequencies of events (in the limit of infinite amounts of such). A more general definition of probabilities is the Bayesians' one, which regards probabilities as quantifiers of statements, expressing how much those should be expected to be true. Although this definition looks subjective on the first glance, it rests on a solid mathematical theorem by Richard Cox6 that any extension of binary logic to a continuum of single valued truthfulness quantifier that respects a minimum of requirements must be isomorphic to probability theory. And this definition turns out to embrace the frequentists' definition in case that infinite numbers of comparable events are actually available and can be counted. The majority of authors of this special issues seem to favor the Bayesian perspective, which is probably a result of our own, the guest editors', preference and not necessarily a reflection of the prevailing position of contemporary scientists. The different views on probabilities of frequentists and Bayesians are addressed by the article of Allen Caldwell.7 He builds a bridge between these antagonistic camps by explaining frequentists' constructions in a Bayesian language. In particular, his argumentation why the omnipresent p-value in statistical and experimental literature might often capture a relevant signal of model misfit is reassuring, given that a large fraction of our medical system is based on decisions backed up by p-values. A Bayesian might add that it would be even more reassuring, if a system rigorously build on logic and its continuous extension would be used for decisions affecting live-or-death questions. The laws governing probabilities themselves seem to reflect that of more elementary mathematical structures. Kevin Knuth shows how these emerge from lattice theory, which deals with relations in partially ordered sets like that of collections of possibilities.8 The sum rule of probability theory is a direct consequence of the underlying lattice structure. Interestingly, a number of other theories are resting also on such lattices and therefore exhibit their own versions of sum rules: measures of sets, mutual information, the Euler characteristics of topological spaces, as well as quantum mechanical interactions that are quantified by pairs of numbers, the real and imaginary parts of complex probability amplitudes (see also4). The theory of information has very practical application in physics and elsewhere. Manfred Opper discusses how the solutions of stochastic differential equations can be inferred via variational methods, which have their origin in thermodynamics.9 Torsten Enßlin introduces information field theory (IFT), the information theory for fields, as a means to infer the configuration of a physical field from incomplete and noisy data.10 The mathematical apparatus for this is that of quantum field theory (QFT), just with ordinary probabilities instead of complex probability amplitudes as in the latter. Could it be that the difference between IFT and QFT is just be the same as between classical information theory and quantum mechanics, the transition from real valued to complex valued quantifier for statements? Anyhow, any practical inference machinery, be it for dynamical systems or for fields, requires sophisticated numerical methods. Michael Betancourt reviews Markov Chain Monte Carlo Methods to calculate expectation values over high dimensional probabilities.11 Theo Steininger and coworkers present the NIFTy3 software package for numerical information field theory calculations, which can handle very high dimensional field inference problems.12 Finally, in an article focused on the subtleties of the Monty Hall problem and mostly aiming at information theoretical entertainment, Torsten Enßlin and Margret Westercamp investigate the possibility that human intuition might be more rational than contemporary psychology thinks.13 A number of possible topics for an issue on physics of information are not addressed here. We would have loved to have articles on Landauer's principle, which states that erasing information has to cost energy, on QBism, a Bayesian version of quantum mechanics, on artificial and biological neural networks, as well as any other aspect of the information-soaked biological world. What has a reader of this special issue to be equipped with in order to follow the lines of thoughts? In addition to the usual physicist's education, the minimal additional equipment are the basics of probability theory and the concept of entropy. Most of the other concepts used in the articles rest on those and are introduced there. Let A, B, and C be propositions (or possibilities). The probability P ( A | B ) of A being true assuming that B is true is a number between 0 and 1, with P ( A | B ) = 1 expressing certainty of A under B ( B ⇒ A ) and P ( A | B ) = 0 expressing certainty of not A being the case under B ( B ⇒ ¬ A ).

Annalen der Physik•Mar 1, 2019•2 citations

Destruction ab Toto: Nothing from Something

Vlatko Vedral

The main view promoted by this book is that underlying many different aspects of reality is some form of information processing. The theory of information started rather innocently, as the result of a very specific question that Shannon considered, which was how to maximize the capacity of communication between two users. Shannon showed that all we need is to associate a probability to an event, and defined a metric that allowed you to quantify the information content of that event. Interestingly, because of its simplicity and intuitiveness, Shannon’s views have been successfully applied to many other problems. We can view biological information through Shannon’s theory as a communication in time (where the objective of natural selection is to propagate the gene pool into the future). But it is not only that communications and biology are trying to optimize information. In physics, systems arrange themselves so that entropy is maximized, and this entropy is quantified in the same way as Shannon’s information. We encounter the same form of information in other phenomena. Financial speculation is also governed by the same concept of entropy, and optimizing your profit is the same problem as optimizing your channel capacity. In social theory, society is governed by its interconnectedness or correlation and this correlation itself is quantified by Shannon’s entropy. Underlying all these phenomena was the classical Boolean logic where events had clear outcomes, either yes or no, on or off, and so on. In our most accurate description of reality, given by quantum theory, we know that bits of information are an approximation to a much more precise concept of qubits. Qubits, unlike bits, can exist in a multitude of states, any combination of yes and no, on and off. Shannon’s information theory has been extended to account for quantum theory and the resulting framework, quantum information theory, has already shown a number of advantages. The greater power of quantum information theory is manifested in more secure cryptographic protocols, a completely new order of computing, quantum teleportation, and a number of other applications that were simply not possible according to Shannon’s view.

Oxford University Press eBooks•Feb 8, 2018•0 citations

Classical Shannon Theory

Mark M. Wilde

We cannot overstate the importance of Shannon's contribution to modern science. His introduction of the field of information theory and his solutions to its two main theorems demonstrate that his ideas on communication were far beyond the other prevailing ideas in this domain around 1948. In this chapter, our aim is to discuss Shannon's two main contributions in a descriptive fashion. The goal of this high-level discussion is to build up the intuition for the problem domain of information theory and to understand the main concepts before we delve into the analogous quantum information-theoretic ideas. We avoid going into deep technical detail in this chapter, leaving such details for later chapters where we formally prove both classical and quantum Shannon-theoretic coding theorems. We do use some mathematics from probability theory, namely, the law of large numbers. We will be delving into the technical details of this chapter's material in later chapters (specifically, Chapters 10, 12, and 13). Once you have reached later chapters that develop some more technical details, it might be helpful to turn back to this chapter to get an overall flavor for the motivation of the development. Data Compression We first discuss the problem of data compression. Those who are familiar with the Internet have used several popular data formats such as JPEG, MPEG, ZIP, GIF, etc. All of these file formats have corresponding algorithms for compressing the output of an information source.

Cambridge University Press eBooks•Apr 18, 2013•0 citations

Review of The Theory of Quantum Information John Watrous

Stephen Fenner

This is an extremely clear, carefully written book that covers the most important results in the sprawling field of quantum information. It is perfect for a reference, self-study, or a graduate course in quantum information. It makes no attempt to be broad or encyclopedic, but instead goes deep into the core topics. The definitions and theorems are all precisely worded, and (starting in Chapter 2) all results have complete proofs, making the book largely self-contained. The book focuses heavily on the mathematical results and nuts-and-bolts techniques underpinning current research, and as such gives the reader a thorough and flexible toolkit for proving new results. If you are just looking for a broad but cursory survey of the field, then this is probably not the book for you. If, however, you want a working knowledge of the core results and proof techniques of quantum information with an eye toward doing cutting-edge research in the field, then this book will be an indispensable addition to your library. The mathematical theory of quantum information studies the ultimate abilities and limits of transmitting and processing information using the laws of quantum mechanics. It owes much of its motivation to classical information theory, which was largely developed by Claude Shannon in the mid 20th century, and to quantum mechanics itself (of course). It addresses basic questions like: how much information can be transmitted through quantum channels, noisy or otherwise, and how entanglement helps. The theory informs, and is informed by, its sister disciplines of quantum computation and quantum communication (which overlap with physics and computer science), although in some sense it is more fundamental. Though he occasionally mentions applications to these other areas, Watrous seats his book squarely in the realm of pure mathematics.

ACM SIGACT News•Mar 16, 2021•0 citations

Classical Shannon Theory

Mark M. Wilde

We cannot overstate the importance of Shannon's contribution to modern science. His introduction of the field of information theory and his solutions to its two main theorems demonstrate that his ideas on communication were far beyond the other prevailing ideas in this domain around 1948. In this chapter, our aim is to discuss Shannon's two main contributions in a descriptive fashion. The goal of this high-level discussion is to build up the intuition for the problem domain of information theory and to understand the main concepts before we delve into the analogous quantum information-theoretic ideas. We avoid going into deep technical detail in this chapter, leaving such details for later chapters where we formally prove both classical and quantum Shannontheoretic coding theorems. We do use some mathematics from probability theory (namely, the law of large numbers). We will be delving into the technical details of this chapter's material in later chapters (specifically, Chapters 10, 13, and 14). Once you have reached later chapters that develop some more technical details, it might be helpful to turn back to this chapter to get an overall flavor for the motivation of the development. Data Compression We first discuss the problem of data compression. Those who are familiar with the Internet have used several popular data formats such as JPEG, MPEG, ZIP, GIF, etc. All of these file formats have corresponding algorithms for compressing the output of an information source. A first glance at the compression problem might lead one to believe that it is not possible to compress the output of the information source to an arbitrarily small size, and Shannon proved that this is the case. This result is the content of Shannon's first noiseless coding theorem. An Example of Data Compression We begin with a simple example that illustrates the concept of an information source. We then develop a scheme for coding this source so that it requires fewer bits to represent its output faithfully. Suppose that Alice is a sender and Bob is a receiver. Suppose further that a noiseless bit channel connects Alice to Bob—a noiseless bit channel is one that transmits information perfectly from sender to receiver, e.g., Bob receives “0” if Alice transmits “0” and Bob receives “1” if Alice transmits “1.”

Cambridge University Press eBooks•Nov 2, 2016•0 citations

Introduction

B. Roy Frieden

Aim of the book The overall aim of this book is to develop a theory of measurement that incorporates the observer into the phenomenon under measurement. By this theory, the observer becomes both a collector of data and an activator of the physical phenomenon that gives rise to the data. These ideas have probably been best stated by J. A. Wheeler (1990), (1994): All things physical are information-theoretic in origin and this is a participatory universe … Observer participancy gives rise to information; and information gives rise to physics. The measurement theory that will be presented is largely, in fact, a quantification of these ideas. However, the reader might be surprised to find that the ‘information’ that is used is not the usual Shannon or Boltzmann entropy measures, but one that is relatively unknown to physicists, that of R. A. Fisher. During the same years that quantum mechanics was being developed by Schroedinger (1926) and others, the field of classical measurement theory was being developed by R. A. Fisher (1922) and co-workers (see Fisher Box, 1978, for a personal view of his professional life). According to classical measurement theory, the quality of any measurement(s) may be specified by a form of information that has come to be called Fisher information.Since these formative years, the two fields - quantum mechanics and classical measurement theory - have enjoyed huge success in their respective domains of application. And until recent times it has been presumed that the two fields are distinct and independent.

Cambridge University Press eBooks•Dec 10, 1998•0 citations
Nihil est in intellectu quod non prius fuerit in sensu. The Physics of Information This book describes the limits for the communication of information with waves. How many ideas can we communicate by writing on a sheet of paper? How well can we hear a concert? How many details can we distinguish in an image? How much data can we get from our internet connection? These are all questions related to the transport of information by waves. Our sensing ability to capture the differences between distinct waveforms dictates the limits of the amount of information that is delivered by a propagating wave. The problem of quantifying this precisely requires a mathematical description and a physical understanding of both the propagation and the communication processes. We focus on the propagation of electromagnetic waves as described by Maxwell's theory of electromagnetism, and on communication as described by Shannon's theory of information. Although our treatment is mostly based on classical field theory, we also consider limiting regimes where the classical theory must give way to discrete quantum formulations. The old question of whether information is physics or mathematics resounds here. Information is certainly described mathematically, but we argue that it also has a definite physical structure. The central theme of this book is that Shannon's information-theoretic limits are natural. They are revealed by observing physical quantities at certain asymptotic scales where finite dimensionality emerges and observational uncertainties are averaged out. These limits are also rigorous, and obey the mathematical rules that govern the model of reality on which the physical theories are based. Shannon's Laws Originally introduced by Claude Elwood Shannon in 1948, and continuing up to its latest developments in multi-user communication networks, information theory describes in the language of mathematics the limits for the communication of information. These limits are operational, of real engineering significance, and independent of the semantic aspects associated with the communication process.
What is information? A short answer could be: Anything that changes our minds and states of thinking! Information is virtual. It can be carried by speech, an image, scratches on a stone, patterns in a photon field, the connections of neurons in our brains or even by the wavefunction of an electron. Information does not depend on the actual means of transmission nor does its identity change when changing the carrier. Information is physical. It needs to be sustained by a physical substrate and rearranging that substrate to contain the wanted information requires to spend energy or work. Without a physical world, information can not be stored, processed, or transmitted. Our world is permeated with different representations of information. Actually Information about the physical laws governing our Universe can be found everywhere and in everything. Could it then be that the real fundamental elements of this world are tiny bits of information? Could we get it from bit?1 A number of researcher investigate this possibility, and several of their attempts to recover physical laws from the principles governing information are reported about in this special issue on physics of information. The articles by Gregg Jaeger,2 by Ariel Caticha,3 and by John Skilling and Kevin Knuth4 address the reconstruction of quantum mechanics from information theory. Furthermore, Kevin Knuth and James Walsh5 show how to recover space-time from particles sending each others signals in an initially geometry-free setting. A central element of most works on information is the concept of a probability. Probabilities can be defined via various routes. Most familiar might be their definitions by the frequentists school as relative frequencies of events (in the limit of infinite amounts of such). A more general definition of probabilities is the Bayesians' one, which regards probabilities as quantifiers of statements, expressing how much those should be expected to be true. Although this definition looks subjective on the first glance, it rests on a solid mathematical theorem by Richard Cox6 that any extension of binary logic to a continuum of single valued truthfulness quantifier that respects a minimum of requirements must be isomorphic to probability theory. And this definition turns out to embrace the frequentists' definition in case that infinite numbers of comparable events are actually available and can be counted. The majority of authors of this special issues seem to favor the Bayesian perspective, which is probably a result of our own, the guest editors', preference and not necessarily a reflection of the prevailing position of contemporary scientists. The different views on probabilities of frequentists and Bayesians are addressed by the article of Allen Caldwell.7 He builds a bridge between these antagonistic camps by explaining frequentists' constructions in a Bayesian language. In particular, his argumentation why the omnipresent p-value in statistical and experimental literature might often capture a relevant signal of model misfit is reassuring, given that a large fraction of our medical system is based on decisions backed up by p-values. A Bayesian might add that it would be even more reassuring, if a system rigorously build on logic and its continuous extension would be used for decisions affecting live-or-death questions. The laws governing probabilities themselves seem to reflect that of more elementary mathematical structures. Kevin Knuth shows how these emerge from lattice theory, which deals with relations in partially ordered sets like that of collections of possibilities.8 The sum rule of probability theory is a direct consequence of the underlying lattice structure. Interestingly, a number of other theories are resting also on such lattices and therefore exhibit their own versions of sum rules: measures of sets, mutual information, the Euler characteristics of topological spaces, as well as quantum mechanical interactions that are quantified by pairs of numbers, the real and imaginary parts of complex probability amplitudes (see also4). The theory of information has very practical application in physics and elsewhere. Manfred Opper discusses how the solutions of stochastic differential equations can be inferred via variational methods, which have their origin in thermodynamics.9 Torsten Enßlin introduces information field theory (IFT), the information theory for fields, as a means to infer the configuration of a physical field from incomplete and noisy data.10 The mathematical apparatus for this is that of quantum field theory (QFT), just with ordinary probabilities instead of complex probability amplitudes as in the latter. Could it be that the difference between IFT and QFT is just be the same as between classical information theory and quantum mechanics, the transition from real valued to complex valued quantifier for statements? Anyhow, any practical inference machinery, be it for dynamical systems or for fields, requires sophisticated numerical methods. Michael Betancourt reviews Markov Chain Monte Carlo Methods to calculate expectation values over high dimensional probabilities.11 Theo Steininger and coworkers present the NIFTy3 software package for numerical information field theory calculations, which can handle very high dimensional field inference problems.12 Finally, in an article focused on the subtleties of the Monty Hall problem and mostly aiming at information theoretical entertainment, Torsten Enßlin and Margret Westercamp investigate the possibility that human intuition might be more rational than contemporary psychology thinks.13 A number of possible topics for an issue on physics of information are not addressed here. We would have loved to have articles on Landauer's principle, which states that erasing information has to cost energy, on QBism, a Bayesian version of quantum mechanics, on artificial and biological neural networks, as well as any other aspect of the information-soaked biological world. What has a reader of this special issue to be equipped with in order to follow the lines of thoughts? In addition to the usual physicist's education, the minimal additional equipment are the basics of probability theory and the concept of entropy. Most of the other concepts used in the articles rest on those and are introduced there. Let A, B, and C be propositions (or possibilities). The probability P ( A | B ) of A being true assuming that B is true is a number between 0 and 1, with P ( A | B ) = 1 expressing certainty of A under B ( B ⇒ A ) and P ( A | B ) = 0 expressing certainty of not A being the case under B ( B ⇒ ¬ A ).
The main view promoted by this book is that underlying many different aspects of reality is some form of information processing. The theory of information started rather innocently, as the result of a very specific question that Shannon considered, which was how to maximize the capacity of communication between two users. Shannon showed that all we need is to associate a probability to an event, and defined a metric that allowed you to quantify the information content of that event. Interestingly, because of its simplicity and intuitiveness, Shannon’s views have been successfully applied to many other problems. We can view biological information through Shannon’s theory as a communication in time (where the objective of natural selection is to propagate the gene pool into the future). But it is not only that communications and biology are trying to optimize information. In physics, systems arrange themselves so that entropy is maximized, and this entropy is quantified in the same way as Shannon’s information. We encounter the same form of information in other phenomena. Financial speculation is also governed by the same concept of entropy, and optimizing your profit is the same problem as optimizing your channel capacity. In social theory, society is governed by its interconnectedness or correlation and this correlation itself is quantified by Shannon’s entropy. Underlying all these phenomena was the classical Boolean logic where events had clear outcomes, either yes or no, on or off, and so on. In our most accurate description of reality, given by quantum theory, we know that bits of information are an approximation to a much more precise concept of qubits. Qubits, unlike bits, can exist in a multitude of states, any combination of yes and no, on and off. Shannon’s information theory has been extended to account for quantum theory and the resulting framework, quantum information theory, has already shown a number of advantages. The greater power of quantum information theory is manifested in more secure cryptographic protocols, a completely new order of computing, quantum teleportation, and a number of other applications that were simply not possible according to Shannon’s view.
We cannot overstate the importance of Shannon's contribution to modern science. His introduction of the field of information theory and his solutions to its two main theorems demonstrate that his ideas on communication were far beyond the other prevailing ideas in this domain around 1948. In this chapter, our aim is to discuss Shannon's two main contributions in a descriptive fashion. The goal of this high-level discussion is to build up the intuition for the problem domain of information theory and to understand the main concepts before we delve into the analogous quantum information-theoretic ideas. We avoid going into deep technical detail in this chapter, leaving such details for later chapters where we formally prove both classical and quantum Shannon-theoretic coding theorems. We do use some mathematics from probability theory, namely, the law of large numbers. We will be delving into the technical details of this chapter's material in later chapters (specifically, Chapters 10, 12, and 13). Once you have reached later chapters that develop some more technical details, it might be helpful to turn back to this chapter to get an overall flavor for the motivation of the development. Data Compression We first discuss the problem of data compression. Those who are familiar with the Internet have used several popular data formats such as JPEG, MPEG, ZIP, GIF, etc. All of these file formats have corresponding algorithms for compressing the output of an information source.
This is an extremely clear, carefully written book that covers the most important results in the sprawling field of quantum information. It is perfect for a reference, self-study, or a graduate course in quantum information. It makes no attempt to be broad or encyclopedic, but instead goes deep into the core topics. The definitions and theorems are all precisely worded, and (starting in Chapter 2) all results have complete proofs, making the book largely self-contained. The book focuses heavily on the mathematical results and nuts-and-bolts techniques underpinning current research, and as such gives the reader a thorough and flexible toolkit for proving new results. If you are just looking for a broad but cursory survey of the field, then this is probably not the book for you. If, however, you want a working knowledge of the core results and proof techniques of quantum information with an eye toward doing cutting-edge research in the field, then this book will be an indispensable addition to your library. The mathematical theory of quantum information studies the ultimate abilities and limits of transmitting and processing information using the laws of quantum mechanics. It owes much of its motivation to classical information theory, which was largely developed by Claude Shannon in the mid 20th century, and to quantum mechanics itself (of course). It addresses basic questions like: how much information can be transmitted through quantum channels, noisy or otherwise, and how entanglement helps. The theory informs, and is informed by, its sister disciplines of quantum computation and quantum communication (which overlap with physics and computer science), although in some sense it is more fundamental. Though he occasionally mentions applications to these other areas, Watrous seats his book squarely in the realm of pure mathematics.
We cannot overstate the importance of Shannon's contribution to modern science. His introduction of the field of information theory and his solutions to its two main theorems demonstrate that his ideas on communication were far beyond the other prevailing ideas in this domain around 1948. In this chapter, our aim is to discuss Shannon's two main contributions in a descriptive fashion. The goal of this high-level discussion is to build up the intuition for the problem domain of information theory and to understand the main concepts before we delve into the analogous quantum information-theoretic ideas. We avoid going into deep technical detail in this chapter, leaving such details for later chapters where we formally prove both classical and quantum Shannontheoretic coding theorems. We do use some mathematics from probability theory (namely, the law of large numbers). We will be delving into the technical details of this chapter's material in later chapters (specifically, Chapters 10, 13, and 14). Once you have reached later chapters that develop some more technical details, it might be helpful to turn back to this chapter to get an overall flavor for the motivation of the development. Data Compression We first discuss the problem of data compression. Those who are familiar with the Internet have used several popular data formats such as JPEG, MPEG, ZIP, GIF, etc. All of these file formats have corresponding algorithms for compressing the output of an information source. A first glance at the compression problem might lead one to believe that it is not possible to compress the output of the information source to an arbitrarily small size, and Shannon proved that this is the case. This result is the content of Shannon's first noiseless coding theorem. An Example of Data Compression We begin with a simple example that illustrates the concept of an information source. We then develop a scheme for coding this source so that it requires fewer bits to represent its output faithfully. Suppose that Alice is a sender and Bob is a receiver. Suppose further that a noiseless bit channel connects Alice to Bob—a noiseless bit channel is one that transmits information perfectly from sender to receiver, e.g., Bob receives “0” if Alice transmits “0” and Bob receives “1” if Alice transmits “1.”
Aim of the book The overall aim of this book is to develop a theory of measurement that incorporates the observer into the phenomenon under measurement. By this theory, the observer becomes both a collector of data and an activator of the physical phenomenon that gives rise to the data. These ideas have probably been best stated by J. A. Wheeler (1990), (1994): All things physical are information-theoretic in origin and this is a participatory universe … Observer participancy gives rise to information; and information gives rise to physics. The measurement theory that will be presented is largely, in fact, a quantification of these ideas. However, the reader might be surprised to find that the ‘information’ that is used is not the usual Shannon or Boltzmann entropy measures, but one that is relatively unknown to physicists, that of R. A. Fisher. During the same years that quantum mechanics was being developed by Schroedinger (1926) and others, the field of classical measurement theory was being developed by R. A. Fisher (1922) and co-workers (see Fisher Box, 1978, for a personal view of his professional life). According to classical measurement theory, the quality of any measurement(s) may be specified by a form of information that has come to be called Fisher information.Since these formative years, the two fields - quantum mechanics and classical measurement theory - have enjoyed huge success in their respective domains of application. And until recent times it has been presumed that the two fields are distinct and independent.