|
2 | 2 | "cells": [ |
3 | 3 | { |
4 | 4 | "cell_type": "code", |
5 | | - "execution_count": 2, |
| 5 | + "execution_count": 1, |
6 | 6 | "metadata": {}, |
7 | | - "outputs": [], |
| 7 | + "outputs": [ |
| 8 | + { |
| 9 | + "name": "stderr", |
| 10 | + "output_type": "stream", |
| 11 | + "text": [ |
| 12 | + "Matplotlib is building the font cache; this may take a moment.\n" |
| 13 | + ] |
| 14 | + } |
| 15 | + ], |
8 | 16 | "source": [ |
9 | 17 | "#math and linear algebra stuff\n", |
10 | 18 | "import numpy as np\n", |
|
21 | 29 | "cell_type": "markdown", |
22 | 30 | "metadata": {}, |
23 | 31 | "source": [ |
24 | | - "# Orthonormalization process\n", |
25 | | - "This notebooks exposes how to perform orthogonalization on a set of vector, such that the resulting method is fast and accurate.\n", |
| 32 | + "# Orthonormalization /whitening process\n", |
| 33 | + "This notebooks exposes how to perform orthogonalization on a set of vector, such that the resulting method is fast and accurate. This topic is mentionned in the notebook \"StatisticalTest\" where we investigate the use of whitening and decorrelation transforms\n", |
26 | 34 | "\n", |
27 | 35 | "## Orthonormalization through Gram-Schmidt\n", |
28 | 36 | "\n", |
|
71 | 79 | "cell_type": "markdown", |
72 | 80 | "metadata": {}, |
73 | 81 | "source": [ |
74 | | - "## Orthogonalization using Gram matrix and Choleski\n", |
| 82 | + "## Orthonormalization using Gram matrix and Choleski\n", |
75 | 83 | "\n", |
76 | 84 | "Another possible solution is based on the Gram matrix $M_G$ of the set of vectors $\\{\\vec{v_0}, \\vec{v_1}, \\dots, \\vec{v_{n-1}} \\},ドル computed as $M_G(i,j)= \\langle \\vec{v_i}, \\vec{v_j} \\rangle$\n", |
77 | 85 | "\n", |
|
95 | 103 | " I &= (V M_G^{-\\frac{1}{2}})^t (V M_G^{-\\frac{1}{2}})\n", |
96 | 104 | "\\end{align*}\n", |
97 | 105 | "\n", |
98 | | - "Such that $(M_G^{\\frac{1}{2}} V)$ is the new orthonormalized set of vector. Now, we are interested in finding, either" |
| 106 | + "Such that $(M_G^{\\frac{1}{2}} V)$ is the new orthonormalized set of vector. Now, we are interested in finding, square root of matrix $M_G,ドル which hopefully is exactly what the Choleski decomposition does. IE from [choleski factorization](https://en.wikipedia.org/wiki/Cholesky_decomposition) we can see that a hermitian positive-definite matrix $A,ドル admits a decomposition of the form $A = LL^{*}$ where L is a lower triangular matrix with real and positive diagonal entries, and $L^{*}$ denotes the conjugate transpose of $L$. Every Hermitian positive-definite matrix (and thus also every real-valued symmetric positive-definite matrix) has a unique Cholesky decomposition." |
99 | 107 | ] |
100 | 108 | }, |
101 | 109 | { |
|
194 | 202 | "Mg2 = np.dot(V2.T,V2)\n", |
195 | 203 | "\n", |
196 | 204 | "#check that Mg2 is orthonormal\n", |
197 | | - "assert( np.allclose(Mg2-Mg2*np.identity(nbVec),np.zeros((nbVec,nbVec))) )\n", |
| 205 | + "assert( np.allclose(Mg2-np.identity(nbVec),np.zeros((nbVec,nbVec))) )\n", |
198 | 206 | "\n", |
199 | 207 | "#Show matrix Mg2\n", |
200 | 208 | "plt.imshow(Mg2,interpolation='none')" |
|
205 | 213 | "metadata": {}, |
206 | 214 | "source": [ |
207 | 215 | "## Remark\n", |
208 | | - "It should be noticed that a Gram matrix is always symmetric positive semi difinite, and as such can be diagonalized in an orthonormal basis, which makes $Q^t$ the inverse of $Q$.\n", |
| 216 | + "It should be noticed that a Gram matrix is always symmetric positive semi definite, and as such can be diagonalized in an orthonormal basis, which makes $Q^t$ the inverse of $Q$.\n", |
209 | 217 | "\n", |
210 | 218 | "In the Choleski case, it should be noticed that computing the inverse of a lower triangular matrix is a low complexity operation. Demonstration sketch is as follow:\n", |
211 | 219 | "Let $L$ be a lower triangular matrix of size n, we have $L=D(I+N),ドル where D is a diagonal matrix, I is the identity matrix, and N is a strictly lower triangular matrix.\n", |
|
220 | 228 | "In the framework of high performance computing, one may be interested in using dtrtri, the level 3 BLAS Triangular Matrix Inversion (TMI)." |
221 | 229 | ] |
222 | 230 | }, |
| 231 | + { |
| 232 | + "cell_type": "markdown", |
| 233 | + "metadata": {}, |
| 234 | + "source": [ |
| 235 | + "### General notes on whitening\n", |
| 236 | + "It might be noticed that if you find a $W_W$ such that the following holds:\n", |
| 237 | + "\\begin{align*}\n", |
| 238 | + " W_W X (W_W X)^T &= W_W X X^T W_W^T \\\\\n", |
| 239 | + " &= W_W C W_W^T \\\\\n", |
| 240 | + " &= I\n", |
| 241 | + "\\end{align*}\n", |
| 242 | + "Then, the property will hold for any $W_W'$ such that $W_W' = A W_W$ with $A$ an arbitrary matrix with unitary property, ie $A^{*}A=I$\n", |
| 243 | + "There might then be multiple choice for the final whitening matrix, and one have some additional freedom to choose the whitenning transform such that it exhibit different sort of optimality, such as zca like transform for instance, that optimize for average cross-covariance or average cross-correlation between original and transformed data.\n", |
| 244 | + "\n", |
| 245 | + "More on this in [Optimal whitening and decorrelation](https://arxiv.org/abs/1512.00809)" |
| 246 | + ] |
| 247 | + }, |
223 | 248 | { |
224 | 249 | "cell_type": "code", |
225 | | - "execution_count": null, |
| 250 | + "execution_count": 3, |
226 | 251 | "metadata": {}, |
227 | 252 | "outputs": [], |
228 | | - "source": [] |
| 253 | + "source": [ |
| 254 | + "from IPython.display import IFrame\n", |
| 255 | + "#IFrame(\"doc/Orthonormalization/optimal_whitening_decorrelation.pdf\", width=1200, height=800)" |
| 256 | + ] |
| 257 | + }, |
| 258 | + { |
| 259 | + "cell_type": "markdown", |
| 260 | + "metadata": {}, |
| 261 | + "source": [ |
| 262 | + "## Geometric interpretation of orthonormalization transforms\n", |
| 263 | + "It is interesting to notice that there is a strong link between orthonormalization transforms, and ellipsoid, as, most on the time, one is interested in factorizing either gram or covariance matrix, which are both hermitian matrices (positive semidefinite )" |
| 264 | + ] |
229 | 265 | }, |
230 | 266 | { |
231 | 267 | "cell_type": "markdown", |
|
241 | 277 | ], |
242 | 278 | "metadata": { |
243 | 279 | "kernelspec": { |
244 | | - "display_name": "Python 3", |
| 280 | + "display_name": "Python 3 (ipykernel)", |
245 | 281 | "language": "python", |
246 | 282 | "name": "python3" |
247 | 283 | }, |
|
255 | 291 | "name": "python", |
256 | 292 | "nbconvert_exporter": "python", |
257 | 293 | "pygments_lexer": "ipython3", |
258 | | - "version": "3.8.10" |
| 294 | + "version": "3.13.1" |
259 | 295 | } |
260 | 296 | }, |
261 | 297 | "nbformat": 4, |
262 | | - "nbformat_minor": 2 |
| 298 | + "nbformat_minor": 4 |
263 | 299 | } |
0 commit comments