I am sure that many of you are familiar with the definition of the second law of thermodynamics. After all, it is recklessly taught in physics 201,
without paying attention to the physical interpretations of the words constituting this well memorized sentence,
In a closed system, entropy always increases.
That's why it might be the most interdisciplinary, but yet the most misunderstood law of nature. So we need to fix it here.
Let's begin with the word
always.
You all know the famous story of the ink drop. When left into a glass of water,
the drop diffuses into the water, exhibiting Brownian motion. We know - because this is
what we observed up until now - that the diffused ink molecules
1 will
never gather back together and form a drop.
Or
will they, if we wait long enough?
Brownian motion simply means
random motion. Molecules move by hitting and pushing each other in a
chaotic environment. Remember the time that the concert is over and you are trying to get out with
your friends, also with a million additional people, simultaneously. In this analogy, you and your
friends form a "friend drop". Even though you don't take a step intentionally, as the crowd begins
to go out, you begin to be pushed by them, and your motion gets out of your control. You begin to
move randomly.
This random motion of molecules were precisely explained by Albert Einstein, in his
1905
paper. What you need to know about it for now is the following : steps that you
are taking when you are trying to get out of the concert hall are
independent of each other.
Remember that your motion is not in control, you are pushed around by other people (in the
ink case, other molecules), and you also push other people when you are being pushed around.
So you don't know where you will set your next step on. But you can not take a step of 1 km long, right?
So intuitively, your hunch tells you that there must be an
underlying probability distribution of your step
size. In the Brownian motion case, these step sizes are (as you may guess) normally distributed.
So each step of yours becomes a normal random variable, and the cumulative sum of these variables
constitute your trajectory. But what determines the parameters of this normal distribution? Keep in mind
that escaping from the concert hall is just an analogy. Molecules don't have an intention to reach to the
exit door, neither they are trying to take a cab home. So in case that they are not being pushed around,
they don't have a reason to move. This gives us the first parameter of our Normal distribution, which is
the mean, being equal to zero.
Back to the concert analogy. What is one of the things that determines the length of your step size? The crowd!
Your step size depends on how crowded the hall is, and from a molecule's perspective, how
dense
the medium is. Of course, density is not the only parameter for a molecule to go faster or slower.
Temperature, viscosity of the fluid, and the size of the molecule itself (how fat and tall you are)
are also important. All these properties combine in a single parameter, which called the
diffusion
coefficient. Physically, diffusion coefficient is the measure of how much molecules diffuse thorough
a unit surface in a unit time at a concentration gradient of unity
[1]. Intuitively, it provides
us a measure of how
fast the molecules diffuse. Since your speed depends on the length of your step
for a fixed distance, diffusion coefficient is significant in determining the variance of the step size,
which is the second parameter of our Normal distribution.
Up until now, we only talked about a single step. But to proceed further, we need to determine the
trajectory of our molecule. Recall that each step is independent of each other, and cumulative sum of
the steps constitute the trajectory. Which means that, in a discrete case, if I am forced to take 10
steps in the crowd (I am not taking them intentionally), my trajectory will be the sum of 10 normally
distributed, zero mean random variables, which is again, a normal random variable with zero mean. Since
these variables are independent of each other, the variance of my trajectory will be equal to the variance
of my each step multiplied by the number of my steps, which in this case is equal to 10. For the case of a continuous
motion, we will adapt these discrete sums as integrals over a time interval.
Enough with the analogy, let's calculate the probability to see our beloved drop back again.
Assume that you have a cylinder glass with a radius of \(R=4\) cm, and it is full of water up to \(h=6\) cm height.
You injected your ink drop into water with an injection syringe, precisely in the middle. You can see the
illustration of this setup in Figure 1.
 |
| Figure 1: Illustration of the diffusion system. |
Let \({\mathbf{X^{i}}} = [X_{1}^{i},X_{2}^{i},X_{3}^{i}]\) denote the position vector of the \(i^{th}\)
ink molecule, and let \(X_{j}^{i}\) denote the position variable in the \(j^{th}\) dimension at
time \(t\) (For the sake of simplicity, variable \(t\) is not included in the notation). Since the glass
is 3-D, we will have \(j=\{1,2,3\}\), and \(i=\{1,2,...,N\}\), where \(N\) is the total number of ink
molecules in our initial drop. Statistically, the average radius of a water drop is equal to 1.5 mm, and
it contains approximately \(10^{20}\) \(H_{2}O\) molecules, so let us use these values for our ink drop too.
Let's denote the radius of the drop with \(r=1.5\)mm, and \(N=10^{20}\). Since the drop is so small, also
let us assume that the initial positions of all molecules are zero, i.e., \({\mathbf{x^{i}}}=[0,0,0]\) for \(i=\{1,2,...,N\}\) at \(t=0\).
Since all the molecules will exhibit Brownian motion, their position can be modelled as a continuous time
stochastic process, in which, the increments in a given dimension are normally distributed, and independent
of each other. As time elapses, these independent increments add up to form molecules' path in the
medium, and as these independent increments add up, the variance of the position adds up too (Recall the
concert hall analogy). So the position variable at time \(t\) becomes normally distributed such that
\begin{align}
X_{1,2,3}^{i} \sim \mathcal{N}(0,\sigma^2) \phantom{00} \text{i=\{1,2,...,N\}},
\label{eq:g}
\end{align}
where \(\sigma^2 = 2Dt\), and \(D\) denotes the diffusion coefficient.
Let us simplify the problem a little. We were trying to find the probability that all \(10^{20}\) molecules form
back the original drop again with a proper alignment, i.e., position of the molecules must not collapse, and
their centers must be located such that they produce a sphere-like shape. Instead of this, let me re-define the
problem. What is the probability that all ink molecules are
in a sphere of radius \(r\), centered at
\({\mathbf{C}}=[C_{1},C_{2},C_{3}]\) at the same instant \(t\)? So for a single ink molecule, probability of being in
the drop at a given time \(t\) and for a given center \(\mathbf{C}\) is equal to
\begin{align}
P\big\{\mathbf{X^{i}} \in \mathcal{D}\mid t, \mathbf{C}\big\} = P\big\{\lVert \mathbf{X^{i}}-\mathbf{C} \rVert \leq +r\mid t, \mathbf{C}\big\}.
\label{eq:ineq}
\end{align}
where \(\mathcal{D}\) denote the connected set of points
that constitutes the drop. But we need to calculate the probability of
all ink molecules being in the drop at the same time. As a result,
since the ink molecules' trajectories are independent of each other, what we need to calculate becomes
\begin{align}
P\big\{\mathbf{X^{1:N}} \in \mathcal{D} \mid t, \mathbf{C} \big\} = \prod_{i=1}^{N}P\big\{\mathbf{X^{i}} \in \mathcal{D} \mid t, \mathbf{C} \big\} = \prod_{i=1}^{N}P\big\{\lVert \mathbf{X^{i}}-\mathbf{C} \rVert \leq +r\mid t, \mathbf{C}\big\}.
\label{eq:prod}
\end{align}
Furthermore, due to the Brownian motion, steps taken at each dimension is also independent of each other. So instead of
calculating the Euclidian distance between the center of the drop and the ink molecules, we can check whether each dimension
of the ink molecules' falls inside the range of the drop in the corresponding dimension. To do that, we need
to be more precise about the center of the drop. Since the drop can not get close the boundaries more than its radius,
center of the drop must be bounded such that
\begin{align}
-(R-r) &\leq c_{1,2} \leq +(R-r),\nonumber \\
-(h/2-r) &\leq c_{3} \leq +(h/2-r).\nonumber \\
\end{align}
Since there is no restriction on where the drop will form, it is safe to assume that the center is
uniformly distributed
between these ranges. As a result, center variables for each dimension become
\begin{align}
C_{1,2} \sim \mathcal{U}(-(R-r),&+(R-r)), \nonumber \\
C_{3} \sim \mathcal{U}(-(h/2-r),&+(h/2-r)). \nonumber
\end{align}
Calculating whether all the molecules' positions fall in the drop's corresponding ranges gives us
\begin{align}
P\big\{\mathbf{X^{1:N}} \in \mathcal{D} \mid t, \mathbf{C} \big\} &= \prod_{i=1}^{N}P\big\{-r \leq X_{1}^{i}-C_{1}\leq +r\mid t, C_{1}\big\} \times \nonumber \\
& P\big\{-r \leq X_{2}^{i}-C_{2}\leq +r\mid t, C_{2}\big\}P\big\{-r \leq X_{3}^{i}-C_{3}\leq +r\mid t, C_{3}\big\}.
\label{eq:prod12}
\end{align}
We need to relax the problem a little bit more here. The calculation given in \eqref{eq:prod12} requires to calculate the probabilities
conditioned on \(\mathbf{C}\).
This makes sense, since at time \(t=0\), we will see the drop at \(\mathbf{C}=[0,0,0]\) with probability \(1\), because
we put it there in the first place.
In order to make these probabilities independent of \(\mathbf{C}\), we need all the molecules diffuse into the medium, and the solution to become
homogeneous. This means that the
ink molecules can be
everywhere with equal probability, which is the asymptotic behaviour of \eqref{eq:g}, i.e., as \(t \to \infty \). As a result, for a homogeneous solution, we
can write
\begin{align}
X_{1,2}^{i} \sim \mathcal{U}(-R,&+R), \nonumber \\
X_{3}^{i} \sim \mathcal{U}(-h/2,&+h/2), \nonumber
\label{eq:g2}
\end{align}
for \({i=\{1,2,...,N\}}\). Only under this assumption, it is not going to matter where the center is, since the ink molecules become randomly distributed in water.
Furthermore, by assuming the solution is homogeneous, all the probability distributions become independent of time too.
Let's assume that enough time had passed, the solution became homogeneous, and the probabilities in \eqref{eq:prod12} became independent of \(\mathbf{C}\) and \(t\).
Now we can re-write \eqref{eq:prod12} such that
\begin{align}
P\big\{\mathbf{X^{1:N}} \in \mathcal{D} \big\} &= \prod_{i=1}^{N}P\big\{-r \leq X_{1}^{i}-C_{1}\leq +r\big\}\nonumber \\
&P\big\{-r \leq X_{2}^{i}-C_{2}\leq +r\big\}P\big\{-r \leq X_{3}^{i}-C_{3}\leq +r\big\}.
\label{eq:prod2}
\end{align}
To calculate \eqref{eq:prod2}, we need to know the probability distribution of \((X^{i}_{j}-C_{j})\) for \(j=\{1,2,3\}\). Since \(C_{j}\)
is symmetric around zero for all \(j\), probability distribution of \((X^{i}_{j}+C_{j})\) will be identical to the probability distribution
of \((X^{i}_{j}-C_{j})\). Let us define a new random variable, \(Z_{j}^{i}\), such that
\begin{align}
Z_{j}^{i} = X^{i}_{j}+C_{j},
\end{align}
which permits us to write the probability distribution function of \(Z_{j}^{i}\) as
\begin{align}
f_{Z^{i}_{j}}(z^{i}_{j}) = f_{C_{j}}(c_{j}) \ast f_{X_{j}^{i}}(x_{j}^{i}).
\end{align}
This is the convolution of two uniform probability distributions. Writing the convolution explicitly yields to the piecewise probability
distribution function
$$ f_{Z^{i}_{1,2}}(z^{i}_{1,2}) =
\left\{
\begin{array}{ll}
\frac{z^{i}_{1,2}+2R-r}{4(R-r)R} & : z^{i}_{1,2} \in [-(2R-r), -r],\\
\frac{1}{2R} & : z^{i}_{1,2} \in [-r, r],\\
\frac{z^{i}_{1,2}-(2R-r)}{4(R-r)R} & : z^{i}_{1,2} \in [r, 2R-r].
\end{array}
\right.$$
$$ f_{Z^{i}_{3}}(z^{i}_{3}) =
\left\{
\begin{array}{ll}
\frac{z^{i}_{3}+h-r}{2(h/2-r)h} & : z^{i}_{3} \in [-(h-r), -r],\\
\frac{1}{h} & : z^{i}_{3} \in [-r, r],\\
\frac{z^{i}_{3}-(h-r)}{2(h/2-r)h} & : z^{i}_{1,2} \in [r, h-r].
\end{array}
\right.$$
which is plotted in Figure 2.
 |
Figure 2:\( \,\,\,f_{Z^{i}_{1,2}}(z^{i}_{1,2})\) (on the left) and \( \,\,\,f_{Z^{i}_{1,2}}(z^{i}_{3})\) (on the right). |
Now we can calculate \eqref{eq:prod2} by integrating \(f_{Z^{i}_{j}}(z^{i}_{j})\) over \([-r, +r]\), which is
\begin{align}
P\Big\{{-r \leq Z_{1,2}^{i} \leq +r}\Big\} &=\int_{-r}^{+r}f_{Z^{i}_{j}}(z^{i}_{1,2})\,dz^{i}_{1,2}, \nonumber \\
&=\int_{-r}^{+r}\frac{1}{2R}\,dz^{i}_{1,2},\nonumber \\
&=\frac{r}{R}.
\end{align}
\begin{align}
P\Big\{{-r \leq Z_{3}^{i} \leq +r}\Big\} &=\int_{-r}^{+r}f_{Z^{i}_{j}}(z^{i}_{3})\,dz^{i}_{3},\nonumber \\
&=\int_{-r}^{+r}\frac{1}{h}\,dz^{i}_{3},\nonumber \\
&=\frac{2r}{h}.
\end{align}
Plugging these results in \eqref{eq:prod2} gives us
\begin{align}
P\big\{\mathbf{X^{1:N}} \in \mathcal{D} \big\} = \Big(\frac{2r^2}{R^2h}\Big)^N.
\label{eq:final}
\end{align}
Equation \eqref{eq:final} tells us that the volume of the solution, which appears to be at the denominator, affects \(P\big\{\mathbf{X^{1:N}} \in \mathcal{D} \big\}\)
by the power of \(N\). In other words, larger the glass, lower the probability of seeing our drop back. Additionally, recall that \(r = 1.5\)mm,
which is a
really small value compared to the dimensions of the solution.
Let's talk in numbers. Recall that we were trying to calculate the probability of
all ink molecules
being
in the drop
simulatenously. Before that, let's focus on calculating the probability of
only one ink molecule
being in the drop, i.e., \(P\big\{\mathbf{X^{i}} \in \mathcal{D}\big\}\). The reason is the
following : If \(P\big\{\mathbf{X^{i}} \in \mathcal{D} \big\}\) is so small, then multiplying it
by itself \(N\) times will be practically zero due to the machine precision. Although we have chosen a realistic number of molecules for
an ink drop \((N=10^{20})\), we must be careful about \(N\), and increase it step by step.
Let's start calculating. You can see the probabilities for different \(N\) values in Table 1. Notice that approximately after \(N=60\), which is
really small compared to the realistic value of \(N=10^{20}\), machine precision becomes inadequate, and probabilities appear to be zero.
Of course these numbers are some kind of an upperbound, since we relaxed our problem in some aspects.
\(\newcommand\T{\Rule{0pt}{1em}{.3em}}\)
\begin{array}{c|c}
\hline N \,\,\, & P\big\{\mathbf{X^{1:N}} \in \mathcal{D} \big\} \T \\\hline
1 & 4.6875\times 10^{-5} \\\hline
2 \T & 2.1973\times 10^{-9} \\\hline
5 \T & 2.2631\times 10^{-22} \\\hline
10 \T & 5.1217\times 10^{-44} \\\hline
15 \T & 1.1591\times 10^{-65} \\\hline
20 \T & 2.6232\times 10^{-87} \\\hline
30 \T & 1.3435\times 10^{-130} \\\hline
50 \T & 3.5242\times 10^{-217} \\\hline
60 \T & 1.8050\times 10^{-260} \\\hline
80 \T & 0 \\\hline
100 \T & 0 \\\hline
\end{array}
|
Table 1: Probabilities of forming a drop containing \(N\) molecules. |
So, what does all those numbers mean?
They mean that second law of thermodynamics is a
statistical law.
They mean that even though the probabilities are
extremely low, there
exists a probability to
see the ink molecules form a drop back. Yes,
practically it is equal to zero; but it's important to be
aware of that
this is just a
practical result. This perspective also helps us to understand the transitions from
microscale to macroscale. In microscale, if you observe only two molecules hitting each other and bouncing back to their
original positions,
you don't think they violate the second law. But if a homogeneous water-ink solution forms a
drop from nowhere, things become
strange. This is because you are not talking about the behaviour of only two
molecules anymore, you are talking about trillions of trillions.
So if everybody is convinced, let's fix the first part of the sentence.
Second law
doesn't say that it is
impossible to see a case such that the entropy
decreases.
Instead, it says that
most of the time (usually very close to
always), systems tend toward their most
probable state
[2],
which is (as calculated above) is obviously
not the state which the entropy
decreases.
References
[1] http://www.thermopedia.com/content/696/
[2] Peter M. Hoffman,
Life's Ratchet : How molecular machines extract order from chaos.
Footnotes
1) Ink is composed of many different components, but let's assume there is an identical "ink molecule".
