Probability Theory Problems

Let's have fun on probability theory, here is my first problem set in the said subject.

Problems

1. It was noted that statisticians who follow the deFinetti school do not accept the Axiom of Countable Additivity, instead adhering to the Axiom of Finite Additivity.
   1. Show that the Axiom of Countable Additivity implies Finite Additivity.
   2. Although, by itself, the Axiom of Finite Additivity does not imply Countable Additivity, suppose we supplement it with the following. Let $A_1\supset A_2\supset\cdots\supset A_n\supset \cdots$ be an infinite sequence of nested sets whose limit is the empty set, which we denote by $A_n\downarrow\emptyset$. Consider the following:
      
      
      Axiom of Continuity: If $A_n\downarrow\emptyset$, then $P(A_n)\rightarrow 0$
      
      Prove that the Axiom of Continuity and the Axiom of Finite Additivity imply Countable Additivity.
2. Prove each of the following statements. (Assume that any conditioning event has positive probability.)
   1. If $P(B)=1$, then $P(A|B)=P(A)$ for any $A$.
   2. If $A\subset B$, then $P(B|A)=1$ and $P(A|B)=P(A)/P(B)$.
   3. If $A$ and $B$ are mutually exclusive, then $$\begin{equation}\nonumber P(A|A\cup B) = \displaystyle\frac{P(A)}{P(A)+P(B)}. \end{equation}$$
   4. $P(A\cap B\cap C)=P(A|B\cap C)P(B|C)P(C)$.
3. Prove that the following functions are cdfs.
   1. $\frac{1}{2}+\frac{1}{\pi}\arctan(x), x\in (-\infty, \infty)$
   2. $(1+e^{-x})^{-1},x\in (-\infty,\infty)$
   3. $e^{-e^{-x}}, x\in (-\infty, \infty)$
   4. $1-e^{-x}, x\in (0,\infty)$
   5. the function defined in (1.5.6), (Check in the reference below.)
4. A cdf $F_X$ is stochastically greater than a cdf $F_{Y}$ if $F_{X}(t)\leq F_{Y}(t)$ for all $t$ and $F_{X}(t) < F_{Y}(t)$ for some $t$. Prove that if $X\sim F_X$ and $Y\sim F_Y$, then $$\begin{equation}\nonumber P(X>t) \geq P(Y>t)\;\text{for every}\;t \end{equation}$$ and $$\begin{equation}\nonumber P(X>t)>P(Y>t),\;\text{for some}\; t \end{equation}$$ that is, $X$ tends to be bigger than $Y$.
5. Let $X$ be a continuous random variable with pdf $f(x)$ and cdf $F(x)$. For a fixed number $x_0$, define the function $$\begin{equation}\nonumber g(x) = \begin{cases} f(x) / [1-F(x_0)]& x \geq x_0\\ 0 & x < x_0. \end{cases} \end{equation}$$ Prove that $g(x)$ is a pdf. (Assume that $F(x_0)<1$.)
6. For each of the following, determine the value of $c$ that makes $f(x)$ a pdf.
   1. $f(x)=\mathrm{c}\sin x, 0 < x < \pi/2$
   2. $f(x)=\mathrm{c}e^{-|x|},-\infty < x < \infty$

Solutions

1. 1. Proof. Let $\mathscr{B}$ be a $\sigma$-algebra and consider $A_1,A_2,\cdots\in \mathscr{B}$ are pairwise disjoint, then by countable additivity $$\begin{equation}\nonumber P\left(\displaystyle\bigcup_{i=1}^{\infty}A_i\right)=\displaystyle\sum_{i=1}^{\infty}P(A_i). \end{equation}$$ Now, $$\begin{equation} \begin{aligned} P\left(\displaystyle\bigcup_{i=1}^{\infty}A_i\right)&= P\left(\displaystyle\bigcup_{i=1}^{n}A_i\cup\displaystyle \bigcup_{i=n+1}^{\infty}A_i\right)\\ &= P\left(\displaystyle\bigcup_{i=1}^{n}A_i\right)+P\left(\displaystyle \bigcup_{i=n+1}^{\infty}A_i\right),\;(\text{since}\;A_i's\;\text{are disjoints})\\ &=P(A_1)+\cdots+P(A_n)+P\left(\displaystyle \bigcup_{i=n+1}^{\infty}A_i\right),\\ &\quad(\text{by finite additivity})\\ &=\displaystyle\sum_{i=1}^{n}P(A_i)+P\left(\displaystyle \bigcup_{i=n+1}^{\infty}A_i\right) \end{aligned}\nonumber \end{equation}$$ Notice that for any $n$, we can consider $P(A_i),\;i>n$ to be empty. Implying $$\begin{equation}\nonumber P\left(\displaystyle\bigcup_{i=n+1}^{\infty}A_i\right)=\displaystyle \sum_{i=n+1}^{\infty}P(A_i)=P(\emptyset)+P(\emptyset)+\cdots, \end{equation}$$ that is, $$\begin{equation}\nonumber \begin{aligned} P\left(\displaystyle\bigcup_{i=1}^{\infty}A_i\right)&= \displaystyle\sum_{i=1}^{n}P(A_i)+\sum_{i=n+1}^{\infty}P(A_i)\\ &=\displaystyle\sum_{i=1}^{n}P(A_i)+P(\emptyset)+P(\emptyset)+\cdots \end{aligned} \end{equation}$$ $\therefore$ countable additivity implies finite additivity.
      $\hspace{12.5cm}\blacksquare$
   2. From (a), we have shown that countable additivity implies finite additivity, i.e., $$\begin{equation} P\left(\displaystyle\bigcup_{i=1}^{\infty}A_i\right)=\displaystyle\sum_{i=1}^{n}P(A_i)+P\left(\displaystyle \bigcup_{i=n+1}^{\infty}A_i\right) \nonumber \end{equation}$$ If we supplement this with the following condition, that $A_1\supset A_2\supset A_3\supset\cdots$. By Axiom of Continuity, $\displaystyle\lim_{n\to \infty}A_n=\emptyset$, and by Monotone Sequential Continuity, $P\left(\displaystyle\lim_{n\to\infty}A_n\right)= \displaystyle\lim_{n\to\infty}P(A_n)=0$. Now we can write $A_1\supset A_2\supset A_3\supset\cdots$ as $$\begin{equation}\nonumber B_k=\bigcup_{i=k}^{\infty}A_i,\;\text{such that}\;B_{k+1}\subset B_k, \text{implying}\; \lim_{k\to\infty}B_k=\emptyset \end{equation}$$ Thus, finite additivity plus axiom of continuity, we have $$\begin{equation}\nonumber \begin{aligned} P\left(\bigcup_{i=1}^{\infty}A_i\right)&=\lim_{n\to\infty}\left( \sum_{i=1}^{n}P(A_i)+P(B_{n+1})\right)\\ &=\lim_{n\to\infty}\left(\sum_{i=1}^{n}P(A_i)\right)+\lim_{n\to\infty} P(B_{n+1})\\ &=\sum_{i=1}^{\infty}P(A_i)+0,\;(\text{by axiom of continuity}). \end{aligned} \end{equation}$$ Implying countable additivity.
      $\hspace{12.5cm}\blacksquare$
2. 1. Proof. If $P(B)=1$, then $P(S)=P(B)=1$. Because $A\subseteq S$, implies $A\subseteq B$. Thus, $A\cap B = A$, and therefore $$\begin{equation}\nonumber P(A|B)=\displaystyle\frac{P(A\cap B)}{P(B)}=\displaystyle\frac{P(A)}{P(B)}=P(A) \end{equation}$$ $\hspace{12.5cm}\blacksquare$
   2. Proof. If $A\subseteq B$ then $$\begin{equation}\nonumber P(B|A)=\displaystyle\frac{P(A\cap B)}{P(A)}=\displaystyle\frac{P(A)}{P(A)}=1 \end{equation}$$ and, $$\begin{equation}\nonumber P(A|B)=\displaystyle\frac{P(A\cap B)}{P(B)}=\displaystyle\frac{P(A)}{P(B)} \end{equation}$$ $\hspace{12.5cm}\blacksquare$
   3. Proof. If $A$ and $B$ are mutually exclusive, then $$\begin{equation} \nonumber \begin{aligned} P(A|A\cup B)&=\displaystyle\frac{P(A\cap (A\cup B))}{P(A\cup B)}\\ &=\displaystyle\frac{P(A)\cup [P(A\cap B)]}{P(A)+ P(B)}\\ &=\displaystyle\frac{P(A)}{P(A)+ P(B)} \end{aligned} \end{equation}$$ $\hspace{12.5cm}\blacksquare$
   4. Proof. Consider, $$\begin{equation}\nonumber P(A|B\cap C)=\displaystyle\frac{P(A\cap B\cap C)}{P(B\cap C)} \end{equation}$$ Hence, $$\begin{equation}\nonumber P(A\cap B\cap C) = P(A|B\cap C)P(B\cap C) \end{equation}$$ Now $P(B\cap C)=P(B|C)P(C)$, therefore $$\begin{equation}\nonumber P(A\cap B\cap C) = P(A|B\cap C)P(B|C)P(C) \end{equation}$$ $\hspace{12.5cm}\blacksquare$
3. $F(x)$ is a cdf if it satisfies the following conditions:
   1. $\displaystyle\lim_{x\to-\infty}F(x)=0$ and $\displaystyle\lim_{x\to\infty}F(x)=1$
   2. $F(x)$ is nondecreasing.
   3. $F(x)$ is right-continuous.
   1. Proof.
      1. $F(x)=\frac{1}{2}+\frac{1}{\pi}\arctan(x), x\in (-\infty, \infty)$
         
         Above figure was generated by the following $\mathrm{\LaTeX}$ codes:
         
         

         	\begin{tikzpicture}[domain = -5:5, yscale = 6, xscale = 0.7]
         	\draw[->] (-5.5,0) -- (5.5,0) node[right] {$x$};
         	\draw[->] (0,-0.5) -- (0,0.5) node[right] {$y$};
         	\foreach \x/\xtext in {-4/-4, -2/-2, 2/2, 4/4}
         	\draw[shift={(\x,0)}] (0pt,2/6pt) -- (0pt,-2/6pt) node[below] {$\xtext$};
         	\foreach \y/\ytext in {-0.4/-0.4, -0.2/-0.2, 0.2/0.2, 0.4/0.4}
         	\draw[shift={(0,\y)}] (2pt,0pt) -- (-2pt,0pt) node[left] {$\ytext$};
         	\draw[color = blue, thick] (0,0) plot[id = arctan] function{1 / 2 + (1 / pi) * atan(x)} node[right] {$F(x)$};
         	\end{tikzpicture}

         view raw 231ps1-1.tex hosted with ❤ by GitHub
         $$\begin{equation}\nonumber \begin{aligned} \displaystyle\lim_{x\to-\infty}F(x)&=\displaystyle\lim_{x\to-\infty} \left(\frac{1}{2}+\frac{1}{\pi}\arctan(x)\right)\\ &=\frac{1}{2}+\frac{1}{\pi}\displaystyle\lim_{x\to-\infty}\left(\arctan(x)\right)\\ &=\frac{1}{2}+\frac{1}{\pi} \left(\frac{-\pi}{2}\right),\;\text{since}\;\displaystyle\lim_{x\to-\frac{\pi}{2}}\frac{\sin(x)}{\cos(x)}=-\infty\\ &=0\\[0.5cm] \displaystyle\lim_{x\to\infty}F(x)&=\displaystyle\lim_{x\to\infty} \left(\frac{1}{2}+\frac{1}{\pi}\arctan(x)\right)\\ &=\frac{1}{2}+\frac{1}{\pi}\displaystyle\lim_{x\to\infty}\left(\arctan(x)\right)\\ &=\frac{1}{2}+\frac{1}{\pi} \left(\frac{\pi}{2}\right),\;\text{since}\;\displaystyle\lim_{x\to\frac{\pi}{2}}\frac{\sin(x)}{\cos(x)}=\infty\\ &=1 \end{aligned} \end{equation}$$
      2. To test if $F(x)$ is nondecreasing, recall in Calculus that, first differentiation of the function tells us if it is decreasing or increasing. In particular, $\frac{dF(x)}{dx}>0$ tells us that the function is increasing in a given interval of $x$. Thus, $$\begin{equation} \nonumber \frac{dF(x)}{dx}=\frac{d}{dx}\left(\frac{1}{2}+\frac{1}{\pi}\arctan(x)\right)=\frac{1}{\pi(1+x^2)} \end{equation}$$ Confirm the above differentiation with Python using sympy module.
         
         

         	import sympy as sy
         	sy.init_printing(use_latex = True)
         	x = sy.symbols('x')
         	sy.diff((1 / 2) + (1 / sy.pi) * sy.atan(x), x)

         view raw 231ps1-6.py hosted with ❤ by GitHub
         Since $x^2$ is always positive for all $x$, thus $\frac{dF(x)}{dx}>0$, implying $F(x)$ is increasing.
      3. $F(x)$ is continuous, implies that $F(x)$ is right-continuous.
      $\hspace{12.5cm}\blacksquare$
   2. Proof.
      1. $F(x)=\displaystyle\frac{1}{1+e^{-x}}, x\in(-\infty,\infty)$
         
         $$\begin{equation}\nonumber \begin{aligned} \displaystyle\lim_{x\to-\infty}F(x)&=\displaystyle\lim_{x\to-\infty} \left(\frac{1}{1+e^{-x}}\right)\\ &=0\\[0.5cm] \displaystyle\lim_{x\to\infty}F(x)&=\displaystyle\lim_{x\to\infty} \left(\frac{1}{1+e^{-x}}\right)\\ &=\displaystyle\lim_{x\to\infty} \left(\frac{1}{1+\frac{1}{e^{x}}}\right)\\ &=1 \end{aligned} \end{equation}$$ Confirm these in Python,
         
         

         	import sympy as sy
         	sy.init_printing(use_latex = True)
         	x = sy.symbols('x')
         	{sy.limit(1 / (1 + sy.exp(-x)), x, -oo), sy.limit(1 / (1 + sy.exp(-x)), x, oo)}

         view raw 231ps1-7.py hosted with ❤ by GitHub
      2. Using the same method we did in (a), we have $$\begin{equation} \nonumber \begin{aligned} \frac{dF(x)}{dx}&=\frac{d}{dx}\left(\displaystyle\frac{1}{1+e^{-x}}\right)\\ &=\frac{e^{-x}}{(1+e^{-x})^2} \end{aligned} \end{equation}$$ $\frac{dF(x)}{dx}=\frac{e^{-x}}{(1+e^{-x})^2}>0,\;\forall\;x\in(-\infty,\infty)$. Thus the function is increasing in the interval of $x$.
      3. $F(x)$ is continuous, implies the function is right-continuous.
      $\hspace{12.5cm}\blacksquare$
   3. Proof.
      1. $F(x)=e^{-e^{-x}}, x\in (-\infty, \infty)$
         
         $$\begin{equation}\nonumber \begin{aligned} \displaystyle\lim_{x\to-\infty}F(x)&=\displaystyle\lim_{x\to-\infty} \left(e^{-e^{-x}}\right)\\ &=\displaystyle\lim_{x\to-\infty} \left(\frac{1}{e^{\frac{1}{e^{x}}}}\right)\\ &=0\\[0.5cm] \displaystyle\lim_{x\to\infty}F(x)&=\displaystyle\lim_{x\to\infty} \left(e^{-e^{-x}}\right)\\ &=\displaystyle\lim_{x\to\infty} \left(\frac{1}{e^{\frac{1}{e^{x}}}}\right)\\ &=1 \end{aligned} \end{equation}$$
      2. Like what we did above, $\frac{dF(x)}{dx}$ is, $$\begin{equation} \nonumber \frac{dF(x)}{dx}=\frac{d}{dx}\left(e^{-e^{-x}}\right)=e^{-x}e^{-e^{-x}}>0 \end{equation}$$ Because $e^{-x}e^{-e^{-x}}>0,\;\forall\; x\in(-\infty,\infty)$. Then we say $F(x)$ is an increasing function in the interval of $x$.
      3. $F(x)$ is continuous, implies that $F(x)$ is right-continuous.
      $\hspace{12.5cm}\blacksquare$
   4. Proof.
      1. $F(x)=1-\displaystyle\frac{1}{e^{x}}, x\in(0,\infty)$
         
         $$\begin{equation}\nonumber \begin{aligned} \displaystyle\lim_{x\to-\infty}F(x)&=\displaystyle \lim_{x\to 0}F(x)=1-\displaystyle\lim_{x\to 0} \left(\frac{1}{e^{x}}\right) =0\\[0.5cm] \displaystyle\lim_{x\to\infty}F(x)&=1- \displaystyle\lim_{x\to\infty} \left(\frac{1}{e^{x}}\right)=1 \end{aligned} \end{equation}$$
      2. $$\begin{equation}\nonumber \frac{dF(x)}{dx}=\frac{d}{dx}\left(1-\frac{1}{e^{x}}\right)=0-(-e^{-x})=\frac{1}{e^{x}} \end{equation}$$ $F(x)$ is an increasing function since $\frac{1}{e^{x}}>0,\;\forall\;x\in(0,\infty)$.
      3. $F(x)$ is right-continuous, since it is continuous.
      $\hspace{12.5cm}\blacksquare$
   5. Proof. The function in Equation (1.5.6) is given by, $$\begin{equation} F_Y(y)=\begin{cases} \displaystyle\frac{1-\varepsilon}{1+e^{-y}}&\text{if}\;y<0,\; \text{for some}\;\varepsilon, 1>\varepsilon>0\\ \varepsilon+\displaystyle\frac{1-\varepsilon}{1+e^{-y}}&\text{if}\;y\geq 0,\;\text{for some}\;\varepsilon, 1>\varepsilon>0 \end{cases}\nonumber \end{equation}$$
      1. $$\begin{equation}\nonumber \begin{aligned} \displaystyle\lim_{y\to-\infty}F_Y(y)&=\displaystyle\lim_{y\to-\infty} \left(\displaystyle\frac{1-\varepsilon}{1+e^{-y}}\right)=\displaystyle\lim_{y\to-\infty} \left(\displaystyle\frac{1-\varepsilon}{1+\frac{1}{e^{y}}}\right)=0\\[0.5cm] \displaystyle\lim_{y\to\infty}F(y)&=\displaystyle\lim_{y\to\infty} \left(\varepsilon+\displaystyle\frac{1-\varepsilon}{1+e^{-y}}\right)=\varepsilon + \displaystyle\lim_{y\to\infty} \left(\displaystyle\frac{1-\varepsilon}{1+\frac{1}{e^{y}}}\right)=1 \end{aligned} \end{equation}$$
      2. For $y<0$, we have $$\begin{equation} \begin{aligned} \frac{d}{dy}\left(\frac{1-\varepsilon}{1+e^{-y}}\right)&=(1-\varepsilon)\frac{d}{dy}\left(\frac{1}{1+e^{-y}}\right)\\ &=(1-\varepsilon)\frac{(1+\varepsilon^{-y})\cdot 0 - 1\cdot e^{-y}(-1)}{(1+e^{-y})^2}\\ &=\frac{(1-\varepsilon)e^{-y}}{(1+e^{-y})^2} \end{aligned}\nonumber \end{equation}$$ $(1-\varepsilon)>0$ since $0<\varepsilon<1$. Thus for all $y < 0$, $\frac{(1-\varepsilon)e^{-y}}{(1+e^{-y})^2}>0$, implying that the function is increasing.
         
         For $y\geq 0$, $$\begin{equation} \begin{aligned} \frac{d}{dy}\left(\varepsilon+\frac{1-\varepsilon}{1+e^{-y}}\right)&=\varepsilon+\frac{(1-\varepsilon)e^{-y}}{(1+e^{-y})^2} \end{aligned}\nonumber \end{equation}$$ The function is increasing since $\varepsilon + \frac{(1-\varepsilon)e^{-y}}{(1+e^{-y})^2}>0$ for all $y\geq 0$.
      3. Since the function is continuous, then the function is right-continuous.
      $\hspace{12.5cm}\blacksquare$
4. Proof. We know that, $$\begin{equation}\nonumber P(X>t)=1-P(X\leq t)=1-F_X(t) \end{equation}$$ and $$\begin{equation}\nonumber P(Y>t)=1-P(Y\leq t)=1-F_Y(t) \end{equation}$$ Hence we have, $$\begin{equation}\nonumber \begin{aligned} P(X>t)=1-F_X(t)\;&\overset{?}{\geq}\;1-F_Y(t)=P(Y>t)\\ \end{aligned} \end{equation}$$ Since $F_X(t)\leq F_Y(t)$, then the difference $1-F_X(t)$ tends to get bigger than $1-F_Y(t)$. Thus for all $t$, $P(X>t)\geq P(X>t)$.
   
   Now if $F_X(t) < F_Y(t)$ for some $t$, then using the same argument above, $P(X>t) > P(X>t)$ for some $t$.
   $\hspace{13.5cm}\blacksquare$
5. Proof. For a function to be a pdf, it has to satisfy the following:
   1. $g(x)\geq 0$ for all $x$; and,
   2. $\displaystyle\int_{-\infty}^{\infty}g(x)\,dx=1$.
   For any arbitrary $x_0$, $F(x_0)<1$. Thus, $g(x)$ is always positive. Now, $$\begin{equation} \begin{aligned} \int_{-\infty}^{\infty}g(x)\,dx&= \int_{-\infty}^{x_0}g(x)\,dx+ \int_{x_0}^{\infty}g(x)\,dx\\ &=\int_{x_0}^{\infty}g(x)\,dx\\ &=\int_{x_0}^{\infty}\frac{f(x)}{(1-F(x_0))}\,dx\\ &=\frac{1}{1-F(x_0)}\int_{x_0}^{\infty}f(x)\,dx\\ &=\frac{1}{1-F(x_0)}[F(\infty)-F(x_0)]\\ &=\frac{1}{1-F(x_0)}[1-F(x_0)]=1,\;\text{since}\;\lim_{x\to \infty}F(x)=1\\ \end{aligned}\nonumber \end{equation}$$ $\hspace{13.5cm}\blacksquare$
6. In order for $f(x)$ to be a pdf, it has to integrate to 1.
   1. $$\begin{equation} \begin{aligned} \int_{-\infty}^{\infty}f(x)&=\int_{0}^{\frac{\pi}{2}}\mathrm{c}\sin x=\displaystyle\left.-(\mathrm{c})\cos x\displaystyle\right\rvert_{0}^{\frac{\pi}{2}}\\ &=-\mathrm{c}\left(\cos\left(\frac{\pi}{2}\right)-\cos(0)\right)\\ &=-\mathrm{c}(0-1)=1\mathrm{c} \end{aligned}\nonumber \end{equation}$$ Hence, $\mathrm{c}$ is 1. Confirm this with python,
      
      

      	import sympy as sy
      	sy.init_printing(use_latex = True)
      	c, x = sy.symbols('c x')
      	sy.solve(sy.integrate(c * sy.sin(x), (x, 0, pi / 2)) - 1, c)

      view raw 231ps1-8.py hosted with ❤ by GitHub
   2. $$\begin{equation} \begin{aligned} \int_{-\infty}^{\infty}f(x)&=\int_{-\infty}^{\infty} \mathrm{c}\,e^{-|x|}\\ &=\mathrm{c}\left(\int_{-\infty}^{0} \,e^{x}\,dx+\int_{0}^{\infty} e^{-x}\,dx\right)\\ &=\mathrm{c}\left[(e^{0}-e^{-\infty})-(e^{-\infty}-e^{0})\right]\\ &=\mathrm{c}(1+1) = 2\mathrm{c} \end{aligned}\nonumber \end{equation}$$ Hence, c is $\frac{1}{2}$. Confirm this with Python,
      
      

      	import sympy as sy
      	from sympy import oo
      	sy.init_printing(use_latex = True)
      	c, x = sy.symbols('c x')
      	sy.solve(sy.integrate(c * sy.exp(-sy.Abs(x)), (x, -oo, oo)) - 1, c)

      view raw 231ps1-9.py hosted with ❤ by GitHub
      
      

Reference

1. Casella, G. and Berger, R.L. (2001). Statistical Inference. Thomson Learning, Inc.