https://towardsdatascience.com/hyperparameter-optimization-in-python-part-2-hyperopt-5f661db91324
드디어 내가 손튜닝을 벗어나는 날이 오다니...감동스럽다..
Hyperparameter Optimization in Python. Part 2: Hyperopt.
In this blog series, I am comparing python HPO libraries. Before reading this post, I would highly advise that you read Part 0…
towardsdatascience.com
https://en.wikipedia.org/wiki/Kernel_density_estimation
Kernel density estimation - Wikipedia
From Wikipedia, the free encyclopedia Estimator Kernel density estimation of 100 normally distributed random numbers using different smoothing bandwidths. In statistics, kernel density estimation (KDE) is the application of kernel smoothing for probability
en.wikipedia.org
kde라고 아는가?? 여기 분야는 공부할 것이 끝도 없다..끝도 없어...
https://en.wikipedia.org/wiki/Q-Gaussian_distribution
q-Gaussian distribution - Wikipedia
From Wikipedia, the free encyclopedia Probability distribution The q-Gaussian is a probability distribution arising from the maximization of the Tsallis entropy under appropriate constraints. It is one example of a Tsallis distribution. The q-Gaussian is a
en.wikipedia.org
이것도 추가~~~ q의 등장!
The distribution is often favored for its heavy tails in comparison to the Gaussian for 1 < q < 3.
HyperOpt로 머리 아프려고 하는 찰나에 Ray를 발견했다.
https://docs.ray.io/en/latest/tune/index.html
Ray Tune: Hyperparameter Tuning — Ray 2.9.0
To run this example, install the following: pip install "ray[tune]". In this quick-start example you minimize a simple function of the form f(x) = a**2 + b, our objective function. The closer a is to zero and the smaller b is, the smaller the total value o
docs.ray.io
https://arxiv.org/pdf/1905.05583.pdf
-> fine tune BERT
Large Batch Training Of Convolutional Networkshttps://towardsdatascience.com/understanding-fixup-initialization-6bf08d41b427
Understanding Fixup initialization
How to train residual networks without normalization layers.
towardsdatascience.com
Fixup initialization --> 이건 또 뭐람?? 이거 다 공부하려면 잘 시간이 없겠는데...(하지만 자야지~)
Proper initialization of weight matrices is extremely important. 아 weight matrix initialization을 말하는 거군.
출처: 위의 글
What is Fixup?
Fixup (fixed-update initialization) is a recent initialization method for ResNets created by Hongyi Zhang, Yann N. Dauphin and Tengyu Ma. In their paper, the authors showed that it’s possible to train a residual network without batch norm layers. What is more, the authors managed to achieve state of the art performance in image classification and machine translation.
만약 resnet에 normalization layers 가 없다면 ???
그러나, 이런 일이 발생 안 된다고 함. (잘 이해가 안 되긴 한데... 밑의 논문 발췌를 보시라)
데이터 증강하기 Slidding Window with stride
https://arxiv.org/pdf/1908.03265
ON THE VARIANCE OF THE ADAPTIVE LEARNING RATE AND BEYOND
Gradient distribution을 아는가???너무 귀엽다..ㅎㅎ
OOF STACKING
'AI' 카테고리의 다른 글
| Rethinking Personalized Ranking at Pinterest: An End-to-EndApproach 리뷰 (0) | 2024.01.23 |
|---|---|
| Of Spiky SVDs and Music Recommendation 리뷰 (0) | 2024.01.16 |
| Feature Enigneering 공부 (0) | 2024.01.12 |
| GLORY 리뷰 (0) | 2024.01.09 |
| 9주차 회고 (0) | 2024.01.05 |
