As to whether the above definition corresponds to or is consistent with the phenomenonological definition of entropy in thermodynamics is another question and that must be dealt with elsewhere.

Each state has an associated value w_i. The total value of this state variable is given by the expression:

W = Σn_iw_i

One important example of the subject matter is the case in which the entities are molecules in a gas and the state variable is energy. However the above material may also be applied to other cases such as when the entities are households and the state variable is wealth.

The constraints on the distribution are that the number of entities in the states is fixed and also the value of the state variable for the ensemble is fixed; i.e.,

Σ n_i = n
Σ n_iw_i = W

Σ n_i/n = Σ p_i = 1
Σ (n_i/n)w_i = Σ p_iw_i = W/n = w

The Determination of the Maximum Entropy Distribution

The problem is now to choose the p_i's so as to

maximize S = -Σp_iln(p_i)
subject to the constraints
Σp_i =1
and
Σp_iw_i = w

The Lagrangian multiplier method may be used to solve this constrained maximization problem. This means we should seek the unconstrained maximum of

S - λ(Σp_i-1) - μ(Σp_iw_i - w)

The first order conditions are:

∂S/∂p_i - λ - μw_i = 0
for all i

Since

∂S/∂p_i = -ln(p_i) -1

ln(p_i) = 1-λ - μw_i
or
p_i = e^1-λe^-μw_i
for all i.

For convenience let e^1-λ=A. Then the first constraint is that

ΣAe^-μw_i
which means that
A = 1/(ΣAe^-μw_i)

The second constraint is then that

(Σw_ie^-μw_i)/(ΣAe^-μw_i) = w

An Important Special Case

The solution for the value of μ which satisfies the above condition in general would require numberical. This is however a special case in which an analytical solution is possible. That is the special case in which the state variable values are equally spaced and start at zero; i.e.,

w_i = hi for some constant h.

Let H(μ) be defined as:

H(μ) = Σ e^-μhi = Σ(e^-μh)ⁱ

H(μ) = 1/(1-e^-μh)

Furthermore

H'(μ) = Σ(-hi)e^-μhi = -Σ(hi)e^-μhi

H(μ) = (1-e^-μh)^-1
it follows that
H'(μ) = -he^-μh(1-e^-μh)^-2

The second constraint for the case that w_i=hi is that

(Σ(hi)e^-μhi)/(Σe^-μhi) = w.

But

(Σ(hi)e^-μhi)/(Σe^-μhi) = -H'(μ)/H(μ)
= he^-μh/(1-e^-μh).

Therefore

he^-μh/(1-e^-μh) = w

Solving this equation for μ gives

μ = ln(1+h/w)/h.

The Limiting Continuous Case

The limit of the above solution for μ as h goes to zero is, by L'Hospital's Rule,

μ = 1/w.

The above result can also be obtained by considering the continuous case; i.e.,

Maximize S = -∫^∞₀p(x)ln(p(x))dx
with respect to p(x)
subject to the constraints
∫^∞₀p(x)dx = 1
and
∫^∞₀xp(x)dx = w

The Maximum Value of Entropy

Since ln(p_i) = ln(A)-μw_i
S* = Σp_i(ln(A)-μw_i)
= ln(A)Σp_i - μΣw_ip_i
= ln(A) - μw

A Simple Extension

There is a simple extension of the above analysis that is worth carrying out. Suppose the minimum value of the state variable is not necessarily zero but instead is some level j. For the discrete case with evenly-spaced states this means that

p_i = Ae^-μhi for all i≥j
or
p_i = Ae^-hje^-μh(i-j)

μ = ln(1 + h/(w-hj))/h
and for the continuous case of h -> 0
μ = 1/(w-hj)

Conclusion

The maximum entropy distribution is the negative exponential distribution where the parameter of the negative exponential is in the limit of continuous values for the state variable the reciprocal of the average level of the state variable for the population. The maximum entropy for this distribution is equal to the logarithm of what the probability density would be at the average level of the state variable.

Although the concept of a maximum entropy distribution can be formally applied to distributions outside of thermal physics those applications are not necessarily of any empirical significance. In thermal physics any interaction with an extremely high probability increases the entropy so in practice the only significant distributions are the ones with maximum entropy. Outside of thermal physics the interactions of the entities may not operate in the same way. In thermal physical phenomena if there is a high concentration of thermal energy; i.e., high temperature, relative to the surroundings that energy gets dispersed. In economics it is common that entities with a relative low level of wealth transfer wealth to entities with a relative high level of wealth as at a rock concert or a sporting event. Nevertheless it is worth noting that the Italian sociologist-economist, Vilfredo Pareto, when he did a study of the distribution of income in three types of economic systems, capitalist, communist and fascist, found the the distributions for the higher levels of income were all of the same form, negative exponential.

HOME PAGE OF applet-magic.com