선형성 linearity
$$ \frac{d}{dx}(af+bg)= af'+bg' $$
곱의 미분 product rule
$$ \frac{d}{dx}f\cdot g=f'\cdot g + f\cdot g' $$
몫의 미분 quotient rule
$$ \frac{d}{dx}\frac{f}{g}=\frac{g\cdot f' - f\cdot g'}{g^2} $$
Chain rule
$$ f(x,y) , x=g(t), y=h(t)\\ \frac{d}{dt} f(g(t), h(t)) = \frac{\partial f}{\partial x}\frac{\partial g}{\partial t} + \frac{\partial f}{\partial y}\frac{\partial y}{\partial t} $$
벡터 내적 미분
$$ \frac{\partial}{\partial q}(q^Tz)=z ~ (q\in\R^d,z\in\R^d) $$
벡터 노름 미분
$$ \nabla_q||q||=\frac{q}{||q||} $$
행렬-벡터 미분
$$ \frac{\partial}{\partial x} (Ax)=A \\ \frac{\partial}{\partial x} (x^TAx)=(A+A^T)x $$
softmax + cross entropy
$$ \frac{\partial L}{\partial z}=\operatorname{softmax}(z)-y $$