Softmax Linear Units

The abstract introduces SoLU (softmax linear unit), an alternative activation function that reportedly increases the fraction of MLP neurons that are human-interpretable with little to no cost to model performance; the authors validate this with randomized and blinded experiments and use the models to explore transformer information processing, noting trade-offs where increased interpretability may hide other features.