Neural networks

Notes

• Neural Networks are great identifying patterns in data. As a classic example, if you wanted to predict housing prices, you could build a data set that maps features about houses (square feet, location, proximity to Caltrain, etc) onto their actual price, and then train a network to recognize the complex relationship between features and pricing. Training happens by feeding the network features, letting it make a guess about the price, and then correcting the guess (backpropagation).
  • Convolutional Neural Networks work similarly, but with images. Instead of giving a CNN discrete features, you'll usually just use the pixels of the image itself. Through a series of layers, the CNN is able to build features itself (traditionally things like edges, corners) and learn patterns in image data. For example, a CNN might be trained on a dataset that maps images onto labels, and learn how to label new images on its own.

Links

• Neural Network from Scratch (Interactive) (HN)
• But what is a Neural Network? | Deep learning, chapter 1 (2017)
• A Neural Network Playground
• A Beginner's Guide To Understanding Convolutional Neural Networks
• Capsule Networks (CapsNets) – Tutorial
• Chris Olah explains neural nets
• How I Shipped a Neural Network on iOS with CoreML, PyTorch, and React Native - Detailed and awesome article.
• Generative Adversarial Networks (GANs) in 50 lines of code (PyTorch)
• Neural Networks, Types, and Functional Programming
• Recurrent Neural Networks lecture by Yoshua Bengio
• Practical Advice for Building Deep Neural Networks
• Differentiable Architecture Search - Code for DARTS: Differentiable Architecture Search paper.
• TensorSpace.js - Neural network 3D visualization framework, build interactive and intuitive model in browsers, support pre-trained deep learning models from TensorFlow, Keras, TensorFlow.js
• UIS-RNN - Library for the Unbounded Interleaved-State Recurrent Neural Network (UIS-RNN) algorithm, corresponding to the paper Fully Supervised Speaker Diarization.
• ONNX - Open Neural Network Exchange. (Scoring ONNX ML Models with Scala)
• DyNet - Dynamic Neural Network Toolkit.
• gonn - Building a simple neural network in Go.
• Neural Ordinary Differential Equations (2018) - Video explanation | Notes
• Neural Network framework in 25 LOC
• Learning and Processing over Networks (2019) - Workshop presented by Michaël Defferrard and Rodrigo Pena at the Applied Machine Learning Days in January 2019.
• The Next Generation of Neural Networks by Geoffrey Hinton (2007)
• Who Invented Backpropagation? (2014)
• Deep Learning in Neural Networks: An Overview (2015)
• Neural Networks (E01: introduction) (2018) - This series is intended as a light introduction to neural networks, with a focus on the task of classifying handwritten digits.
• Machine Learning for Beginners: An Introduction to Neural Networks (2019)
• A Recipe for Training Neural Networks (2019)
• Exploring Neural Networks with Activation Atlases (2019)
• Curated list of neural architecture search and related resources
• Weight Agnostic Neural Networks (2019) (HN)
• Geoffrey Hinton explains the evolution of neural networks (2019)
• Evolved Turing neural networks
• Intelligent Machinery paper by Alan Turing
• SRU - Training RNNs as Fast as CNNs.
• ODIN - Out-of-Distribution Detector for Neural Networks.
• Ask HN: What Neural Networks/Deep Learning Books Should I Read? (2019)
• Python vs Rust for Neural Networks (2019) (HN)
• Exploring Weight Agnostic Neural Networks (2019) (HN)
• Neural Networks, Types, and Functional Programming (2015)
• Glow - Compiler for Neural Network hardware accelerators.
• Go Neural Net Sandbox - Sandbox for personal experimentation in Go neural net training and Go AI.
• layer - Neural network inference the Unix way.
• XNNPACK - Highly optimized library of floating-point neural network inference operators for ARM, WebAssembly, and x86 (SSE2 level) platforms.
• LSTM implementation explained (2015)
• The Neural Process Family - Contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CNPs), Neural Processes (NPs), Attentive Neural Processes (ANPs).
• Notes on Neural Nets
• RNNoise - Recurrent neural network for audio noise reduction.
• Hacking Neural Networks - Short introduction on methods that use neural networks in an offensive manner.
• Distilling knowledge from Neural Networks to build smaller and faster models (2019)
• Neural Network Processing Neural Networks: An efficient way to learn higher order functions (2019)
• Building a neural net from scratch in Go (2017)
• SparseConvNet - Spatially-sparse convolutional neural network.
• Norse - Library to do deep learning with spiking neural networks.
• Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes (2019) (Code)
• Single Headed Attention RNN: Stop Thinking With Your Head (2019) (HN)
• Visualizing the Loss Landscape of Neural Nets
• primitiv - Neural Network Toolkit.
• On the Relationship between Self-Attention and Convolutional Layers (2019) (Code) (Reddit)
• Implementation of a deep learning library in Futhark
• Single Headed Attention RNN: Stop Thinking With Your Head (2019)
• Using neural networks to solve advanced mathematics equations (2020)
• AlphaFold - Provides an implementation of the contact prediction network, associated model weights and CASP13 dataset as published in Nature. (Paper)
• Go Perceptron - Single / multi layer / recurrent neural network written in Golang.
• Temperature Scaling - Simple way to calibrate your neural network.
• Recurrent Geometric Networks for end-to-end differentiable learning of protein structure
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence (2020) (Tweet)
• kapre - Keras Audio Preprocessors.
• Putting An End to End-to-End: Gradient-Isolated Learning of Representations (2019)
• Memory-Augmented Neural Networks for Machine Translation (2019)
• Have there been any important developments on content addressable memory since hopfield network? (neural networkish) (2020)
• G-Bert - Pre-training of Graph Augmented Transformers for Medication Recommendation.
• Two strange useless things to do with neural nets: a demonstration
• Understanding the Neural Tangent Kernel (2019)
• Cutting out the Middle-Man: Training and Evaluating Energy-Based Models without Sampling (Tweet)
• Haiku - JAX-based neural network library.
• Convolution in one dimension for neural networks (2020)
• Lucid - Collection of infrastructure and tools for research in neural network interpretability.
• Minkowski Engine - Auto-diff neural network library for high-dimensional sparse tensors.
• Generating MIDI melody from lyrics using LSTM-GANs (HN)
• Zoom In: An Introduction to Circuits (2020)
• Lagrangian Neural Networks (2020) (HN)
• Neural Tangents - Fast and Easy Infinite Neural Networks in Python.
• A Survey of Long-Term Context in Transformers (2020)
• OpenNMT-py - Open Source Neural Machine Translation in PyTorch.
• Deep Learning for Symbolic Mathematics (2019) (Paper)
• Google Brain AutoML
• Physics Informed Neural Networks - Data-driven Solutions and Discovery of Nonlinear Partial Differential Equations.
• An introduction to Bayesian neural networks (2020)
• PyTorch Neural Turing Machine
• PyTorch Neural Turing Machine 2
• Learning DAGs with Continuous Optimization (2020)
• Early Vision (2020) - Guided tour of the first five layers of InceptionV1, taxonomized into “neuron groups.”.
• micrograd - Tiny autograd engine and a neural net library on top of it, potentially for educational purposes.
• MiniGrad - Minimal implementation of reverse-mode automatic differentiation (a.k.a. autograd / backpropagation) in pure Python.
• Learning from Small Neural Networks (2020)
• Neural Game Engine - Code to reproduce Neural Game Engine experiments and pre-trained models.
• Graph Convolutional Neural Network Approach to Antibiotic Discovery (2020) (HN)
• KPNNs - Knowledge-primed neural networks.
• ResNeSt - Split-Attention Network.
• Shortcut Learning in Deep Neural Networks (2020)
• Discourse-Aware Attention Model for Abstractive Summarization of Long Documents
• Perovskite neural trees (2020) (HN)
• RigNet: Neural Rigging for Articulated Characters (2020) (Reddit)
• Convolutional neural networks for artistic style transfer (2017)
• Certifiable Robustness to adversarial Attacks; What is the Point? | Nick Frosst (2020)
• LAG: Latent Adversarial Generator
• Towards improved generalization in few-shot classification (2019)
• Convolutional Neural Networks in One Dimension
• Neural Network Pruning (2020)
• Hyperbolic RNN in PyTorch
• deeplearn-rs - Deep learning in Rust.
• Neural networks trained to communicate with each other without any training data
• Classical Piano Composer - Allows you to train a neural network to generate midi music files that make use of a single instrument.
• Weight Standardization - Normalization method to accelerate micro-batch training.
• Teaching Machines to Draw (2017) (In action)
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• pix2code - Generating Code from a Graphical User Interface Screenshot.
• Gated Linear Networks (2019) (HN)
• Curve Detectors (2020)
• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
• Neural Networks and Deep Learning - What they are and how they work.
• Teaching physics to neural networks removes 'chaos blindness' (2020) (HN)
• Understanding Convolutional Neural Networks (Code) (HN)
• Business Card Neural Network (2020)
• Functional Neural Networks (2020)
• Attention Is All You Need (2017)
• Getting Artificial Neural Networks Closer to Animal Brains (2020)
• Foolbox Native - Python toolbox to create adversarial examples that fool neural networks in PyTorch, TensorFlow, and JAX.
• Genann - Minimal, well-tested library for training and using feedforward artificial neural networks (ANN) in C.
• High-Frequency Component Helps Explain the Generalization of Convolutional Neural Networks (2020)
• Awesome Pruning - Curated list of neural network pruning resources.
• Hopfield Networks is All You Need (2020) (Code) (Article) (HN)
• Sparse Networks from Scratch: Faster Training without Losing Performance (2019)
• Jigsaw Labs - Learn Neural Nets
• Implementing a Neural Network in C (Code)
• Web Neural Network API - Dedicated low-level API for neural network inference hardware acceleration. Polyfill
• Clarifying exceptions and visualizing tensor operations in deep learning code (2020)
• Tensor Sensor - Generate more helpful exception messages for numpy/pytorch matrix algebra expressions. (Tweet)
• Explaining RNNs without neural networks (2020)
• A visual explanation for regularization of linear models (2020)
• A Guide to Deep Learning and Neural Networks (2020)
• Handwriting Synthesis - Handwriting Synthesis with RNNs.
• How DeepMind learns physics simulators with Graph Networks (w/ author interview) (2020)
• Build Your Own Artificial Neural Network. It’s Easy! (2020)
• Neural Circuit Policies Enabling Auditable Autonomy
• FermiNet: Fermionic Neural Networks (Quantum Physics and Chemistry from First Principles (2020)) (Tweet)
• What is the Role of a Neuron?
• Marabou - SMT-based tool that can answer queries about a network’s properties by transforming these queries into constraint satisfaction problems.
• Demonstration of the attention mechanism with some toy experiments and explanations
• Augerino - Codebase for Learning Invariances in Neural Networks.
• ELI5 - Python package which helps to debug machine learning classifiers and explain their predictions.
• Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search
• Brian2 - Free, open source simulator for spiking neural networks. (Web)
• Neural Networks from Scratch in Python
• Naszilla - Python library for neural architecture search (NAS).
• The Unreasonable Syntactic Expressivity of RNNs (2020)
• DeepMath Conference - Conference on the Mathematical Theory of DNN's. (HN)
• Coding a Neural Network: A Beginner's Guide (2020)
• Eiffel2 - Neural Network architecture Visualization tool.
• Graph Convolutional Neural Networks (GCNN) models
• Elegy - Neural Networks framework based on Jax inspired by Keras and Haiku.
• SpinalNet - Deep Neural Network with Gradual Input.
• Deeply-supervised Nets
• diagNNose - Python library that facilitates a broad set of tools for analysing hidden activations of neural models.
• MiniSom - Minimalistic implementation of the Self Organizing Maps.
• Basics of Convolution (2020)
• DeepGCNs: Can GCNs Go as Deep as CNNs?
• Tinn - 200 line dependency free neural network library written in C99.
• musicnn - Set of pre-trained musically motivated convolutional neural networks for music audio tagging.
• Convolution Is Fancy Multiplication (HN)
• Tools to Design or Visualize Architecture of Neural Network
• ENNUI - Elegant Neural Network User Interface. (Code)
• Dynamic Graph CNN for Learning on Point Clouds
• robustness - Library for experimenting with, training and evaluating neural networks, with a focus on adversarial robustness.
• JAX, M.D. - Accelerated, Differentiable, Molecular Dynamics. (Paper)
• Neural Reverse Engineering of Stripped Binaries using Augmented Control Flow Graphs
• NN SVG - Generate publication-ready NN-architecture schematics. (HN)
• e3nn - Modular framework for neural networks with Euclidean symmetry.
• Delve - Python package for visualizing deep learning model training.
• Graph Mining @ NeurIPS 2020 (Talks)
• jax_verify - Neural network verification in JAX.
• Self-supervised learning through the eyes of a child (2020) (Code)
• Object-based attention for spatio-temporal reasoning: Outperforming neuro-symbolic models with flexible distributed architectures (2020)
• Soft Threshold Weight Reparameterization for Learnable Sparsity (2020) (Code)
• Understanding the Difficulty of Training Transformers (2020) (Code)
• Awesome Implicit Neural Models
• Edit-distance as objective function papers - Curated list of papers dedicated to edit-distance as objective function.
• SuPar - Collection of state-of-the-art models for Dependency Parsing, Constituency Parsing and Semantic Dependency Parsing.
• Drawing early-bird tickets: Towards more efficient training of deep networks (2020) (Code)
• CountNet: Speaker Count Estimation using Deep Neural Networks
• Applications of Deep Neural Networks Course (2021) (Code)
• DDSL: Deep Differential Simplex Layer for Neural Networks
• Making sense of sensory input (2021)
• char-rnn - Multi-layer Recurrent Neural Networks (LSTM, GRU, RNN) for character-level language models in Torch.
• Awesome Equivariant Networks
• Named Tensor Notation (Code)
• Applications of Deep Neural Networks v2 (2020) (HN)
• Make Your Own Neural Network Blog
• Make Your Own Neural Network Book (Code)
• Representation Learning for Attributed Multiplex Heterogeneous Network (2019) (Code)
• Awesome VAEs - Curated list of awesome work on VAEs, disentanglement, representation learning, and generative models.
• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting (2020) (Code)
• Training Neural Networks is ER-complete (2021)
• Geoff Hinton 2021 – How to represent part-whole hierarchies in a neural network (HN)
• Neural Network Matrix Visualization (2021)
• Multimodal Neurons in Artificial Neural Networks (2021) (HN) (Code)
• OpenAI Microscope - Collection of visualizations of every significant layer and neuron of several common “model organisms” which are often studied in interpretability.
• Real time Interactive Visualization of Convolutional Neural Networks in Unity
• Techniques for Reducing Overfitting (2021) (Tweet)
• Accelerating Neural Networks on Mobile and Web with Sparse Inference (2021)
• Quantization for Neural Networks (2020)
• Introduction to Automatic Hyperparameter Tuning
• Neural Networks Block Movement Pruning
• Torch-Dreams - Making neural networks more interpretable, for research and art.
• Are Deep Neural Networks Dramatically Overfitted? (2019) (HN)
• NASLib - Neural Architecture Search (NAS) library for facilitating NAS research for the community by providing interfaces to several state-of-the-art NAS search spaces and optimizers.
• Neural Network Visualization - Visualization of neural network architectures and parameters.
• Restricted Boltzmann Machine in Haskell
• X-Transformers - Simple but complete full-attention transformer with a set of promising experimental features from various papers. (HN)
• Introduction to Attention Mechanism (2021)
• Understanding Positional Encoding in Transformers (2021)
• Measuring XAI methods with Infidelity and Sensitivity (2021)
• Quasi-Recurrent Neural Networks (2016) (Code)
• deepdream.c - Experiment trying to implement Convolutional Neural Network inference and back-propagation using a minimal subset of C89 language and standard library features.
• Adapting Neural Networks for the Estimation of Treatment Effects (2018) (Code)
• Constructions in combinatorics via neural networks (2021) (Code)
• Artificial Neural Nets Finally Yield Clues to How Brains Learn (2021)
• Neural Additive Models: Interpretable Machine Learning with Neural Nets (2020) (Code)
• Neural-Backed Decision Trees (Code)
• Introduction to Neural Network Verification Book
• ERAN - ETH Robustness Analyzer for Deep Neural Networks.
• What are Transformer Neural Networks? (2021)
• Thinking Like Transformers (2021) (HN)
• TIL: Convolutional Filters Are Weights (2017)
• Solving Mixed Integer Programs Using Neural Networks (2020) (Tweet)
• What Are Convolutional Neural Networks? (2021)
• Fooling Neural Networks (HN)
• Introduction to Neural Network Verification (2021)
• Explainable neural networks that simulate reasoning (2021)
• Evolving Neural Networks through Augmenting Topologies (Code)
• Minimal, clean example of lstm neural network training in python, for learning purposes
• What nice mathematical results there are about neural networks? (2021)
• Temporal and Object Quantification Networks (2021) (Code)
• Encoding Events for Neural Networks (2021)
• Telestrations Neural Networks (2020)
• NN-SVG - Publication-ready NN-architecture schematics. (Web)
• Scientists built deep neural networks that can map between infinite dimensional spaces (2021)
• CNN Explainer - Interpreting Convolutional Neural Networks (2021)
• Transformers from Scratch (2019)
• Transformers from Scratch (2021) (HN)
• General and Scalable Parallelization for Neural Networks (2021)
• 8 Types of Activation Functions in Neural Networks (2021)
• Building a Neural Network in Go (2021)
• Echo State Networks in Python
• Neural Networks for Inference, Inference for Neural Networks (2019)
• Let's Play Distill: Building Blocks (2019) (Tweet)
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017) (Code)
• Neural Network From Scratch (2022) (HN)
• Noether’s Theorem, Symmetries, and Invariant Neural Networks
• CNNs and Equivariance - Part 1/2
• Building a Neural Network in Pure Lisp without Built-in Numbers using only Atoms and Lists (2022)
• Neural Methods in Simulation-Based Inference (2022)
• Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments (2021)
• Computer scientists prove why bigger neural networks do better (2022) (HN)
• On the Difficulty of Extrapolation with NN Scaling (2022)
• Depth Estimation by Convolutional Neural Networks (2016) (Code)
• Generative Flow Networks (2022)
• Feature Learning in Infinite-Width Neural Networks (2021) (Code)
• µTransfer: A technique for hyperparameter tuning of enormous neural networks (2022)
• Restoring and attributing ancient texts using deep neural networks
• Reproducing Yann LeCun 1989 paper "Backpropagation Applied to Handwritten Zip Code Recognition"
• Deep Neural Nets: 33 years ago and 33 years from now (2022) (Reddit) (HN)
• ONNC - Open Neural Network Compiler. (Web)
• Provably robust neural networks - Method for training neural networks that are provably robust to adversarial attacks.
• Rust + WebAssembly + Neural Network
• Neural Networks are not the only universal approximators, so why are they so uniquely effective? (2022)
• Awesome Spiking Neural Networks
• Transformers for software engineers (2022) (HN)
• Exploring Neural Networks Visually in the Browser (2022)
• Neural Network Visualization in the Browser - Neural network library written from scratch in Rust along with a web-based application for building + training neural networks + visualizing their outputs. (Code)
• Sharpened Cosine Similarity - Alternative to convolution for neural networks.
• Google AI Blog: Controlling Neural Networks with Rule Representations (2022)
• Epistemic Neural Networks - Library for uncertainty representation and training in neural networks.
• Transformer in Triton - Implementation of a Transformer, but completely in Triton.
• This AI Does Not Exist - Generate realistic descriptions of made-up machine learning models. (Code)
• Perplexity - Language is a notational semantic for documenting neural networks through diagrams.
• Meta-AF: Meta-Learning for Adaptive Filters
• Sequence Transduction with Recurrent Neural Networks (2021) (Code)
• Papers and Codes for the deep learning in hyperbolic space
• Friends don’t let friends train small diffusion models (2022) (HN)
• Physicists are building neural networks out of vibrations, voltages and lasers (2022) (HN)
• Techniques for Training Large Neural Networks (2022) (HN)
• How fast can we perform a forward pass? (2022) - How fast can you run a transformer model? (Tweet)
• Neural Network Loss Landscapes: What do we know? (2021) (HN)
• Logic Through the Lens of Neural Networks