First, the activation function (AcTIvaTIon FuncTIon)
In order for the neural network to learn complex decision boundaries, we apply a nonlinear activation function at some of its layers. Commonly used functions are sigmoid, tanh, ReLU (RecTIfied Linear Unit linear correction unit) and variants of these functions.
Second, Adadelta
Adadelta is a gradient-based learning algorithm that adjusts the learning rate of each parameter over time. It is more sensitive than hyperparameters and may reduce learning rates. Adadelta is similar to rmsprop and can be used in place of vanilla SGD.
Paper: Adadelta: An Adaptive Learning Rate Approach
Third, Adagrad
Adagrad is an adaptive learning rate algorithm that tracks the squared gradient over time and automatically adapts to the learning rate of each parameter. It can be used instead of vanillaSGD (#sgd), which is especially useful for sparse data, which allows higher learning rates to be assigned to less frequently updated parameters.
Paper: Adaptive Subgradient Method for Online Learning and Stochastic Optimization
Fourth, Adam
Adam is an adaptive learning rate algorithm similar to rmsprop, which is directly estimated by using the running average of the first and second moments of the gradient and has a bias correction function.
Paper: Adam: A Stochastic Optimization Method
Five, affine layer (Affine Layer)
This is a fully connected layer in the neural network. Affine means that each neuron in the previous layer is connected to every neuron in the current layer. In many ways, this is the "standard" layer of neural networks. The affine layer is usually added to the top layer before the final prediction is made by the convolutional neural network or the recurrent neural network. The general form of the affine layer is y = f(Wx + b), where x is the layer input, w is the parameter, b is a deviation vector, and f is a nonlinear activation function.
Attention Mechanism
The attention mechanism is inspired by human visual attention and is an ability to focus on specific parts of the image. Attention mechanisms can be integrated into the architecture of language processing and image recognition to help e-learning "focus" on what to expect when making predictions.
Seven, Alexnet
Alexnet is the name of a convolutional neural network architecture that won a huge advantage in the 2012 ILSVRC Challenge, which led to a renewed focus on convolutional neural networks (CNN) for image recognition. It consists of 5 convolutional layers. Some of these are followed by a max-pooling layer and three fully connected layers of softmax (1000-way softmax) with the final 1000 paths. Alexnet was introduced into the ImageNet classification using deep convolutional neural networks.
Eight, self-encoder (Autoencoder)
A self-encoder is a neural network model whose goal is to predict the input itself, which is usually achieved by a "bottleneck" somewhere in the network. By introducing a bottleneck, the network learns to input a lower dimensional representation, thereby compressing the input into a good representation. Self-encoders are related to dimensionality reduction techniques such as PCA, but because of their nonlinear nature, they can learn more complex mappings. There are a number of widely available self-encoders, including Denoising Autoencoders, Variational Autoencoders, and Sequence Autoencoders.
Noise Reduction Self Encoder Paper:
Stacked Denoising Autoencoders: Learning Useful Representationsin a Deep Network with a Local Denoising Criterion
Changed from the encoder paper:
Auto-Encoding Variational Bayes
Sequence self-encoder paper:
Semi-supervised Sequence Learning
Nine, Average Pooling (Average-Pooling)
Average pooling is a pooling technique for image recognition in convolutional neural networks. Its principle is to slide a window (such as a pixel) on a local area of ​​the feature, and then take the average of all the values ​​in the window. It compresses the input representation into a lower dimensional representation.
X. Backpropagation
Backpropagation is an algorithm used in neural networks to efficiently calculate gradients, or a feedforward computational graph. It can be summed up as a chain rule that applies differentiation from the network output and then propagates the gradient backwards.
paper:
Learning representations by back-propagating errors
XI. Backward propagation through time BPTT: BackpropagationThrough Time
Plug-in Piece Sensor
Feyvan Electronics designs and manufactures NTC temperature sensors, probes, and cable assemblies with excellent long-term stability, high accuracy and short response time in high-temperature sensing applications such as automotive, home appliance and industrial use from -40℃ to +250℃.
With more than 15 years of NTC thermistors and sensor probes production experiences, Feyvan electronics provide various choices for a wide range of applications and are available in custom engineered probe package configurations for a variety of mounting and connectivity options with low costs.
Plug in Sensor,Sensor Coffee Maker,Sensor Coffee Machines,Automotive Sensors
Feyvan Electronics Technology Co., Ltd. , https://www.fv-cable-assembly.com