BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//CMSA - ECPv6.18.2//NONSGML v1.0//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
X-ORIGINAL-URL:https://live-hu-cmsa-222.pantheonsite.io
X-WR-CALDESC:Events for CMSA
REFRESH-INTERVAL;VALUE=DURATION:PT1H
X-Robots-Tag:noindex
X-PUBLISHED-TTL:PT1H
BEGIN:VTIMEZONE
TZID:America/New_York
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20250309T070000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20251102T060000
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20260308T070000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20261101T060000
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20270314T070000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20271107T060000
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20261111T140000
DTEND;TZID=America/New_York:20261111T150000
DTSTAMP:20261112T161630Z
CREATED:20261002T130921Z
LAST-MODIFIED:20261112T161630Z
UID:10004053-1794405600-1794409200@live-hu-cmsa-222.pantheonsite.io
SUMMARY:New Technologies in Mathematics Seminar
DESCRIPTION:New Technologies in Mathematics Seminar \nSpeaker: Johannes von Oswald (Google) \nTitle: Retrieval-Centric Deep Learning: Replacing Weight Matrices with Vector Databases \nAbstract: The deep learning models powering our current AI revolution are built on parameter matrices that form linear layers. These essential components dictate a learning system’s capacity\, as well as its computational and memory requirements during training and inference. In this talk\, I will first highlight that linear layers trained by gradient descent can be expressed as key-value memory systems\, storing inputs as keys and their corresponding output gradients as values (Irie et al.\, 2022). This creates a direct connection between gradient-descent-trained linear layers and popular sequence layers that learn in-context\, such as Mamba or Gated DeltaNet. Consequently\, this duality motivates a gradient-descent-based linear layer counterpart to softmax self-attention. Second\, leveraging this duality\, I will present both historically overlooked and novel “growing” retrieval-centric layers. These provide a principled drop-in replacement for standard linear layers by storing (roughly) inputs and output gradients in a database\, recalling them using non-linear attention. Finally\, I will share ongoing empirical results on language modeling that highlight the practical challenges and promising future directions of this new paradigm in retrieval-centric deep learning. \n 
URL:https://live-hu-cmsa-222.pantheonsite.io/event/newtech_111126/
LOCATION:Virtual
CATEGORIES:New Technologies in Mathematics Seminar
END:VEVENT
END:VCALENDAR