The 34th Conference on Neural Information Processing Systems (NeurIPS) wrapped up last week, 12 Dec. It highlights the most significant achievements in the field of machine learning in the year. The seven day event started with industry exhibitions, which were followed by tutorials, keynote lectures, spotlight talks, posters, and demonstrations. It concluded with specialized workshops and the competition tracks. The conference was held as a purely virtual event for the first time this year and attracted over 23,000 participants from all over the world. IARAI is proud to have taken part in NeurIPS as a Platinum sponsor, hosting high-profile fireside chats, and leading the Traffic4cast sessions as organizer in the NeurIPS Competition Track.
The key NeurIPS events of IARAI included:
1. The Traffic4cast competition track
The results of the Traffic4cast 2020 core competition were summarized at the NeurIPS competition track. IARAI’s founding director Sepp Hochreiter gave an introduction to the data analysis challenge. IARAI’s founding director Michael Kopp outlined the competition design and the compilation of the unique data, provided by HERE Technologies as data sponsor. IARAI’s founding director David Kreil discussed the statistical significance of the top-rank positions in the competition leaderboard. The three winners of the competition and two teams selected by the Traffic4cast Scientific Committee presented the highlight talks. The session concluded with an award ceremony, where the winners were presented with the sought after prizes and the future Traffic4cast competitions were announced.
2. The Traffic4cast Special Session
This session featured an in-depth discussion of the Traffic4cast 2020 competition results, where the winners and four teams selected by the Traffic4cast Scientific Committee presented their solutions in detail. The session was opened by Sepp Hochreiter (IARAI) introducing this year’s multi-modal traffic map movie forecasting challenge. The first keynote talk kicked off the morning session with the provocative topic “The End of Traffic and the Future of Access.” David Levinson (University of Sydney) concentrated on the revolutionary trends in the transport landscape, including the emergence of electric and autonomous vehicles, the changing traffic demands, and traffic programming designed to benefit the travelers collectively. Moritz Neun (IARAI, HERE Technologies) discussed competition design and data format, with the privacy-preserving spatial aggregation of vehicle probes into 0.001 degrees grid cells, and data encoding in static and dynamic channels. The afternoon session was opened by Razvan Pascanu (Deep Mind) with the keynote “Neural networks struggle to transfer.” The discussion covered the latest ideas on how to overcome fundamental limitations in machine learning on non-stationary data, including transfer of learning to new tasks, credit assignment in continual learning, catastrophic forgetting, task interference, and many more. Michael Kopp (IARAI) concluded the session discussing key challenges, including transfer learning and the loss metric, and the next steps in the field of traffic research. Watch the recording of the Traffic4cast Special Session again online!
3. The Fireside chat “Hopfield Networks in 2021”
The fireside chat between Sepp Hochreiter (IARAI) and Dmitry Krotov (IBM Research) focused on their recent groundbreaking research on dense associative memories and modern Hopfield networks and its impact on understanding Transformer architectures and improving attention mechanisms. The panelists discussed integrating modern Hopfield layers as memory modules to enrich deep learning architectures, to distinguish and interpolate between patterns and increase their interpretability. Watch the recording of the fireside chat “Hopfield Networks in 2021” again online!
4. The Fireside chat “Performers and Memory”
The fireside chat between Sepp Hochreiter (IARAI), Krzysztof Choromanski (Google Research) and Johannes Brandstetter (JKU Linz) was inspired by the recent paper “Rethinking Attention with Performers”. This work introduced the Performer, a new model with a linear attention mechanism that approximates Transformer architectures and significantly improves their space and time complexity. Sepp Hochreiter and his team discovered a connection between Transformer’s self-attention and Modern Hopfield networks and showed that it can be extended towards Performers. Watch again the recording of the fireside chat “Performers and Memory”.
5. The Conference paper and spotlight talk
The paper “Modern Hopfield Networks and Attention for Immune Repertoire Classification” by Sepp Hochreiter’s team was accepted to the Advances in Neural Information Processing Systems 33 pre-proceedings and was presented as a spotlight talk at the conference. This work exploits high storage capacity of the new modern continuous Hopfield network introduced in the recent groundbreaking paper “Hopfield Networks is All You Need”. The authors applied the new model to classify the immune status of an individual based on a very large set of immune receptors presented as amino acid sequences.
6. The Ask Me Anything (AMA)
At the IARAI AMA session, IARAI founding directors David Kreil, Michael Kopp, and Sepp Hochreiter held an informal discussion of what it is like working at the institute. The hosts gave an introduction to the institute, discussed its research directions and career and collaboration opportunities, and answered numerous questions from the audience. Watch again the IARAI AMA session.
The institute is proud to have been at the centre of scientific exchange at this prestigious conference. We make recordings of all events available to the public – please see www.iarai.ac.at.
