Well, we can get started, I guess. Um, yeah, we're very happy to have Mariel here.
Who tells us about…
Um, AI-enabled fundamental physics research. Mariel is just starting in Wisconsin?
Finishing my first year. Finishing your first year. Just started. As a new faculty in Wisconsin.
And also splitting your time with Flatiron Institute? Yeah, okay.
And, um, and yep, we're very excited to hear more about this work. Thanks.
Thank you again for the invitation. I was here maybe 2 years ago, I don't know if I met any of you on that visit.
So, um…
Yeah, my talk today is, uh, speculating about, you know, a future, maybe oncoming era of what… how AI will change the way that we do physics analyses.
Um, with the caveat that I'm not going to talk about agents, that's another, um…
rabbit hole. Um…
So, uh, a metaphor that I like to give when I… when I talk about sort of how I think about physics analysis is also thinking about how humans orient themselves in the world.
So I… I like the symbolism of thinking about
you know, not just physics data, but how do we orient ourselves around a city?
Um…
So, for instance, New York, when I first moved here a couple years ago, um,
It's a very overwhelming place, and I think any new city is like this, where it's kind of too much to sort of encapsulate all
it all in one image, or one idea.
So, when you move anywhere new, of course, the first thing you do is just sort of look in your neighborhood and see what's around. You look in your immediate surroundings, and you understand
Where do you get bagels? Where do you get pizza? Where do you do your laundry? Where do you get groceries?
But, okay, I maybe mastered this. This is not my block. I maybe mastered this after, like, a month or something of moving to a new city, but this is not to say that I understand how the city that I live in really operates. Clearly, I need, like, some other layers to pick this apart.
So, um, a critical error, uh, to understanding New York, of course, is the subway system, and
walking around your neighborhood, you might not know that this is, like, a critical length through which you should understand the city. It defines everything that's growing above the subway system as well.
And, um, we don't have to stop here, so, you know, maybe after, like, a couple months of living in a city, you understand your neighborhood, you understand your subway.
Maybe you also want to understand the history of the city that you live in, and by sort of placing a different layer onto your map of a historical context of the city,
like this one, I love, um, that sort of illustrates the wall of Wall Street, um, back when New York was New Amsterdam.
It gives you this, like, different perspective on the city that you live in that really informs your depth of your knowledge.
And maybe you don't want to think about humans at all in this, uh, in this city. Maybe you actually just want to think about, like, the geologic framework of your city and how…
different deposits of different types of rocks informed what kinds of buildings could be built on top of them, and then that shapes what humans came to those areas.
So, you can go back much further in geologic time to kind of understand the place that you live in.
Um, and then, of course, you know, maybe there's more critical maps, like the best pizza in the city that you live in, or, you know, things that are very personal, but, you know, no less important to your understanding of the place that you live in. People can disagree about what the… For sure. Um, yeah, I think quite a lot is missing from this list.
So, anyway, I like this metaphor in thinking about, you know, what are we doing as scientists, thinking of us all as living in this one big city that is the universe, that is, like, way too complex for any single image or map to really capture, and so instead, what we're doing as scientists is
Taking very particular lenses to render different maps of the universe at different scales.
Uh, with different limitations.
And then our job is to, like,
tell the story of stitching all of these different layers together to understand how they inform one another.
And so this is kind of what I think about when I am imagining in my ideal scenario, how would computational tools
bolster this type of analysis or this type of study of, like, mapping in a big picture sense.
So, um, what I'm going to talk about today is sort of imagining what it would take to build a wide-ranging, multi-layered, and data-driven representation of our universe.
that is meant to complement, as well as deepen our existing scientific understanding. So, you know, I won't claim that this data-driven…
perspective is going to be, um, a replacement for our understanding, or that it'll be complete.
But it doesn't have to be complete to be useful, I think, and I think that that's true of maps also.
Um, so, uh, this is a little bit vague, maybe, so be… to be more specific, what I'll talk about is…
by wide-ranging, I'm thinking about multi-detector analyses, or even multidisciplinary analyses, like putting many datasets in conversation with each other.
multi-layered, I'm talking about modalities of information, um, which I'll define later on in the talk.
And then, um, data-driven, I mostly mean training on real data without labels.
To take advantage of just the huge amount and volume of data that we have.
Um, and so to train
AI models without labels, we turn to representation learning or self-supervised learning.
So the… the result of all of this, um, you could call it a foundation model, and maybe you've heard of this metaphor
for things like ChatGPT as a blurry JPEG of the web.
Um, so it's kind of like a blurry, condensed representation of something like the internet.
And I really like this metaphor for a lot of large language models, and so I think we can draw from that in thinking about
the potential of foundation models in science.
That, you know, this blurry JPEG of the web, it is not the internet. It's a distinct thing. But that doesn't mean it's not useful, and so in capturing kind of the essential patterns of the internet,
We can interact with it differently than we can with the body of information of the internet. And it's…
something in that dynamic, that relationship between the two, um, where it finds utility.
Cool, so, um, we have a lot of foundation models. Many of us probably use them
Frequently. Um, so the question is kind of, like, why would this be a particularly unique conversation to be happening within the sciences?
If we already have foundation models that are widely available. So, uh, even though my talk is framed around this idea of, like, what would it take to build a foundation model, why would we want it?
Um, I actually think a lot of my talk is just, like, framed around questions of data, like, what makes physics data special?
And what may or may not, um, present challenges for the existing framework of how we build
machine learning foundation models, um, in the world.
So, we… even though we have…
Oh, yeah.
I'm sorry. I have a question. So I just want to make sure I'm hearing this right. You're using the word foundation model. Are you going to define what that means?
Sure, yeah, kind of intentionally in this talk, I guess I, um…
I'm not framing this as, like, a pitch only for foundation models, but I guess what I would define it as is…
a large-scale machine learning model that's been trained on diverse datasets.
in a self-supervised way, meaning without labels, so it's kind of using the data itself to form its own labels.
And the goal is not to solve any one particular task, but the goal is to create a useful representation
that can translate onto future tasks that we did not necessarily define ahead of time. So that it has some even small amount of emergent properties.
such that it can generalize easily to new tasks.
Or even new datasets that the model hasn't seen before.
Um, but I think maybe just as useful would be thinking of, um,
multi-layered, uh, data representations? Like, how do we put, um,
sources of data from physics into conversation with each other.
Hopefully that answers your question, and if not, we can talk more about it later.
Um…
Okay, okay. Yes, thank you. Right.
break. So, we've… we see a lot of examples of foundation models for… for text or language.
For images, for video,
out in the world right now, and so…
the core question that I'm interested in is basically to what extent will physics data, um, fit within these pre-existing methods that we have to build foundation models for these other, more common data types?
Or to what extent will physics data present, kind of, um…
unique challenges that only physicists will have to, um, step up and present solutions for.
So, I'll frame my talk.
With, um…
three ways in which physics data
has some qualities that I argue are relatively underexplored in standard machine learning settings.
So the… the first point that I want to make is
just zooming in on one type of data that I happen to care about, which is particle physics data.
And so I'll talk about, um, some recent examples that I've worked on, thinking about, uh, how to design machine learning methods for particular qualities of particle physics data.
So, um, I think maybe the most obvious, um,
uh… uh…
quality that makes particle physics data special, I think, is the… the…
grounding in the standard model of particle physics.
It's such a unique resource. It's so, um, it's so powerful and profound.
And, uh, a core part of the standard model of particle physics is, um, is built on symmetry groups, right?
So, um, so we know that symmetries are critical in understanding particle physics.
We also know that machine learning models are actually pretty good at understanding how to represent certain symmetries. So, for instance, image models, by design, have symmetric properties included, so things like translation invariance is, um, kind of naturally encoded into how
convolutional neural nets, which are designed for images.
are trained. So we know that it can handle some types of symmetries, it being machine learning.
Um, we also know that we can do this for even more complicated-looking symmetries that we care about within physics. So, for instance, we could imagine encoding some of the other symmetry groups of the standard model, and we have a framework to build group equivariant convolutional networks, and so you can sort of imagine adapting neural network frameworks for any number of, uh,
symmetry groups that you might care about.
At the same time, um, sort of more recent evidence over the last few years, um, to me suggests that the benefits of equivariant networks, so neural networks that explicitly build in symmetric invariance to certain groups that you care about,
It's really only super beneficial in a data-limited regime.
And that's what these two plots are kind of showing, um, is that even though a symmetry-invariant network
Um, or an equivariant network, uh, is, uh, kind of more accurate, even across higher Lorentz boosts.
Um, most of the accuracy results are reported when you only have a fraction of your original training data. So if you're in a setting where you only have less than half a percent,
of your original training data, maybe you can really see some strong, uh, benefits.
But today, we live in a pretty data-rich regime, and so, to me, the benefits are less clear.
And I think that this is just an interesting example to start out with, because I might naively expect that symmetries should be, like, the most critical piece, um, to consider when we build, uh, models that represent physics data, especially particle physics.
But already, it's… it's clear to me that maybe my intuition is not, um, totally aligned with
the computational needs of representing physics data.
So, I've… yeah, I find this something…
Sorry, so what is in your, in your graph, what is beta?
Oh, this…
If I read, how should I read that graph? As I more data is going to the left? Is that the idea?
It's, um, it's the strength of the Lorentz boost, so it's sort of how extreme
of a shift is applied.
Oh, I…
Yeah, so going from left to right, the Lorenz boost is getting larger, and so the accuracy, um, decreases for a fully connected network that does not have any symmetry awareness built in.
But the plot is at which training percentage?
100%, or…? It's a great question, I don't remember. Um, yeah, I'd have to check.
And then from the table, we're supposed to take away what? As the training percentage goes up, the gap between the two narrows in some sense, or what? I think, really, I would just note that, you know, this is how…
these results had to be reported, is… the comparison is really only useful
for under 5% of the training percentage. Otherwise, the results are compatible, or are the equivariant network is outperformed by the unconstrained flow. Okay.
Right. You said that, but I'm trying to understand how that's being illustrated by the graph or the table.
that symmetry. I mean it intuitively what you said is very, very clear that if you have symmetry and only a little bit of data, then the symmetry is going to be powerful. If you have lots and lots of data, then who needs the symmetry? But I'm just trying to understand how that's illustrated in the graph or the table.
The table, I, like, I have an artificially cut off the table. These are the results that are presented in this paper.
And, uh, with training fractions that are larger, um,
Uh, they…
the Lorentznet is outperformed by the ParticleNet. So here, uh, the accuracy is higher.
Um, but ParticleNet wins out, ultimately, with full access to the data.
Um, but you're right that I don't… I don't show the reverse situation here, because I don't extend the… the table further.
So, within a certain regime, which I argue is, you know, in a smaller data regime, um,
we do see these benefits, but in a larger data regime, uh, the benefits are less clear.
I think for the sake of the audience, maybe the particle net is the unconstrained… it doesn't use… That's right, Lorentznet, right, has sort of Lorentz equinariates, um, constructed into its architecture.
Great. So, uh, I'm just gonna keep…
Breezing ahead, but I invite questions.
throughout, and we can talk more at the end, too.
Um, my point here is basically, um,
it's not always obvious what the best representation of a given dataset is.
And so, arguably, you could think of our choices as sort of spanning, uh…
a, um, a spectrum between maybe fully prescribed industry standard formats that are friendly for image models or sequential models, or point cloud models to ingest.
Um, maybe in the middle is something like a physics-informed representation, where we do choose to build in specific symmetries that are not present in industry standard datasets, such as Lorentz equivariance.
And then maybe on the furthest extreme is, like, a fully self-supervised, unconstrained, you know, we let the model determine what representation is most useful.
in a fully data-driven format.
So, um, it…
I don't think there's a single answer right now to which of these representations is most useful.
It used to be that maybe we had more industry standard formats for our physics data, um,
a few years ago, maybe 10 years ago,
Nowadays, I think more researchers are exploring, kind of, more unconstrained and self-supervised representations for physics. On the previous slide…
Were you comparing the physics formed to the left one or the right one.
On the previous slide, um,
This, uh, this plot is showing the unconstrained model.
Yeah, so is that data-driven, or is that industry standard?
Um, you could say that this is, uh, more of an industry standard representation, and then this would be an equivariant representation.
Um, yes, right.
So, I think my point here is just that the question is not resolved yet. We have some evidence that it seems to be most effective
in a data-limited regime, I'm not counting out the usefulness of building in
Um, physical symmetric constraints into our representations.
But its intention with this other, uh, clear trend, which is that scaling up our models
also yields very powerful representations of our models, and so I think it's, um…
challenging a lot of these intuitions, and I think these two facts will continue to be in tension over the next few years.
Um, next, I want to quickly also mention another application of machine learning, sort of,
capturing particularities of physics data.
And this is related to the idea of what anomalies look like in particle physics.
Um, and so I, uh, I presented, um, slides like this a couple years ago, but I wanted to mention it because it was from a collaboration, um, that David and I had a few years ago.
Um, so, thinking about, um,
When I talk to statisticians about anomalies,
sometimes I realize that we're actually talking about pretty different things. You know, I'm not talking about a one-off.
anomaly. Um, I'm thinking instead of statistical anomalies or group anomalies,
Um, because particle physicists care about, you know, aggregating more and more data, and then showing that our results are incompatible with the background hypothesis.
So, if we care about kind of looking for bump-like anomalies in particles, in particle physics, our machine learning methods can also be designed to reflect this type of search.
And, um, we see this, uh, in a number of more recent applications of machine learning and particle physics to do things like rediscover the Upsilon particle in CMS open data.
using model-agnostic anomaly detection search strategies with machine learning.
And, uh, also for Beyond the Standard Model searches, um, this is a search for a SUSI signal, um, and, uh,
We've shown that, you know, you can inject much smaller amounts of signal, and you can still be sensitive to beyond the standard model signals.
With these types of machine learning anomaly detection strategies.
And, um, uh, what David and I worked on is, um, thinking also about where else in the… in physics do group anomalies show up.
And, uh, realizing that in astrophysics data, and particularly in the form of stellar streams,
Um, you can cast this in a similar light of thinking of group anomalies, um,
in… in, uh, the Milky Way.
And so, um, what we worked on is, sort of,
Uh, thinking about stellar streams, which are these ancient remnants of smaller galaxies or dwarf galaxies,
Um, that have been stretched out due to gravitational forces as they orbit the Milky Way.
And, uh, nevertheless, even though they spatially are very, uh, spread out and quite faint and thin,
Um, they all came from the same astrophysical source.
And so, as a result, they… the stars that are members of the stellar stream just have very similar velocities.
And so if you project into proper motion space,
the Stellar stream, um, velocities really stand out as kind of their own bump compared to the background of other stars that don't appear to be moving in the same way.
And, uh, so we showed that you can sort of cast
this same search for stellar streams in the same way that you would cast a model-agnostic search for particles against a smoothly falling distribution.
And, um, as a result, our machine learning model, which was designed to detect these types of pumps, um, automatically,
can, uh, reproduce, um,
can get very similar, um, can identify similar stars as in sort of a more, um,
Uh… uh…
a more hand-drawn, maybe, or, um, curated list of stars that should be members of the stream.
And, um, and David has extended these studies much further in recent years, too.
Um, so, uh…
So I just wanted to, you know, start out by just sort of listing, like, outside of…
the idea of combining multiple datasets together, just within a particular analysis or study, already we've seen that physics data has particular qualities that are kind of unusual in other settings.
And, um, sometimes treating those qualities carefully
creates, uh, real benefits.
Sometimes the benefits are not as obvious, um, with a careful treatment.
So, next I'm going to imagine
let's say we, uh, were fully convinced that it was actually useful to train a model on multiple datasets, um, from physics.
Even if we wanted to do it, um, what would some of the potential challenges be in combining datasets from multiple areas or multiple detectors?
So, the first thing I'll talk about here is just that, of course, physics data comes in all kinds of formats that are
not obvious, um, to combine.
So, whether we're talking about particle physics data from collider physics, um, we could talk about fluid dynamics, or PDEs,
We could talk about LIGO data, um, or other sources of astrophysics data. All of these, you know, came from very specialized detectors, specialized formats, um, vastly different scales.
Uh, it's not at all clear how we might sort of, um,
have these datasets talk to one another in the context of a machine learning model.
So, um, particle physicists have already thought about this a little bit, um, because we are especially interested in using our data and combining it
or, sorry, comparing it with theoretical predictions.
So, already we have some methods in particle physics to publish our data by removing… and remove detector distortions. So the idea being that we know that our detectors have some biases built into them, so maybe at minimum, if we were to publish our data and have it be, um,
Combined with other sources, we would want to remove those detector biases before we use the data.
So in particle physics, the way that this works is through a process called unfolding, which is also, like, deconvolution. Um, it's basically removing the detector distortions from your data.
And the core idea of it, um, in sort of cartoon terms, is that you start with your measured distribution of some parameter that you care about, but we know that this is not actually the true
Sage at momentum distribution that we cared about, and our detector modified this distribution in some way.
Um, as the information traveled through our detector, which has, um, inefficiencies and smearing effects and, uh, these types of distortions.
So, traditionally, um, the way that we think about, uh, uh, sort of removing these effects is in a kind of re-weighting scenario, where you might start with your measured distribution, and you want to learn some re-weighting,
So that way you can make your measured distribution look more like a true distribution.
And this requires some knowledge of how your detector operates, so that way you have, kind of, samples of
distorted data and, um, true data. So usually it's done with simulation.
And, um, a nice way to do this type of re-weighting is to learn a likelihood ratio. So you can take the probability distributions of each of these two, the measured and the true, um, observable.
And, uh, by learning the likelihood ratio, this gives you a nice re-weighting function, so for any point along the spectrum, you can re-weight the distribution to look more like the true measured.
Uh, data.
And as a quick aside, um, the way that machine learning is involved in this process, um,
is exploiting this fact, which is that a
uh, sort of very vanilla, um, uh, traditional neural network.
that classifies between two classes can often be trained, say, with binary cross-entropy loss.
Which looks like this.
And, uh, the function that ends up minimizing this expression
will satisfy your following for any small variation.
Because it's a minimum of this loss functional.
And from this, we can just, um, kind of extract, uh, this central term and, uh, rearrange, um,
some, uh, the expression to find, uh, this really useful expression. So what I'm showing here is basically on the left is a rescaling of the classifier output that has learned how to optimally classify between two classes, A and B.
And so F of X is, like, the function that defines the classification score.
Um, so you can just kind of take your classifier output and divide it by 1 minus the classifier score.
And on the right, um, we have our likelihood ratio.
um… likelihood ratio here on the right, which is the parameter that we… or, sorry, the function that we wanted to learn in order to learn our re-weighting function in order to remove detector effects.
Okay, so, um, so in practice, the way that we use neural networks is to train classifiers, and then we
sort of reuse the classifiers in order to get these likelihood ratios.
So you might be able to do this for, um, say, two, not just one, but maybe two different observables at once.
And, um, in practice, this is how a lot of LHC measurements are published, um, as…
differential cross-sections at the particle level, meaning they have been sort of corrected for any detector effects.
And what I'm showing here is… is really… it's a two-dimensional measurement, even though it looks like six plots, um, just because it's… it's a measurement of one observable.
in bins of another observable. So you can think of it as just two dimensions, but it's in… it's in bins.
Um, so this is sort of, uh, in a more standard way, how a lot of standard model measurements are reported, um, to correct for these distortions.
Um, but by using machine learning in this framework that I just described, um, we've recently been able to do this for much higher dimensional datasets. So, um, this…
measurement that I'm describing now is a 24-dimensional measurement, whereas previously this was really only able to be done for, like, 2 or 3 observables at a time.
And, uh, so this is what it looks like in cartoon form. You know, we're sort of estimating these likelihood ratios, and we're applying them onto 24 different observables.
such that, as a distribution, all 24 of these observables are corrected to look more like true
true data.
Um, and the non-cartoon version looks like this, where we can sort of report the corrected, um, measurements.
And compare them, the pink and the blue here are, um…
Monte Carlo simulations, and so we can really try to… try to look and see where does our Monte Carlo coverage…
look good, and where does it, um, fall short of our corrected data?
And another nice feature of this?
is that, um, we don't have to be limited to just the 24 observables that we happen to choose for this measurement, so you can take the ratio of two of them,
And, um, this is done because it's using machine learning, but it's done in an unbitten way, and so it allows us to sort of, um, trivially change the bins here to allow
For, um, multiple observables to be compatible.
And further, maybe you want to change the phase space of your measurement. This is something that would not be easy to do with a sort of fixed binned measurement, but because it's unbbinned,
We can apply these types of cuts after the measurement has happened, and then, um, uh,
compare our Monte Carlo in new regimes.
And we can do, sort of, fancier combinations. This is a delta R distribution, which again was not part of the original measurement, but it can be constructed post hoc.
Um, using a combination of around 8 different observables.
Or we can report the average of one observable in bins of another, and so even though all of these are binned,
plots. It's an unbinned measurement, and so I'm just choosing my bins, um, and I could trivially change them.
Point being, we can choose to publish our data in much more flexible formats.
with, uh, corrections for detector effects.
And, uh, this is not something that's only happened at the LHC in recent years, even though the Atlas measurement was a major one.
So, more recently, uh, I led this white paper that's, uh, like a practical guide to unbinned unfolding across a series of experiments.
So, we've had, um, measurements do this from ATLAS, from CMS, from H1, LHCB star, and T2K.
And so, the point being that now this is a, um…
Uh, it's becoming a more popular standardized format for how to think about publishing our data in particle physics.
Such that, um, it's easier for theorists to work with, it's easier for other experimentalists to work with.
And I argue, if we ever wanted to put these datasets in conversation with one another by training a machine learning model on multiple datasets,
using this type of format and correcting for detector effects makes that easier.
Um, and then one other point on this idea of, like, how… how do we have to configure our data in order to
um, potentially enable combinations of multiple datasets.
is that, um, not only do we have to worry about detector biases, like detector-specific effects in our data,
But we also have to think about the representations of our data in a machine learning model.
These are not obvious, um, and…
Uh, right now, it's sort of the common paradigm for how to do this is to create custom embeddings for every single dataset that you might want to train a model on.
So, um, maybe you have…
A model that takes in jets, um, particle jets. You can train your own tokenizer that sort of creates a representation of particle jets.
And, um, you have to sort of verify that the tokenizer has captured all the qualities that you care about of your data.
And then if you want to train a model on, um, galaxy spectra, you have to do the same thing. So you'd have to design your own tokenizer and, like, carefully verify that it captures the properties that you care about.
So we can do this. It's a little tedious, and it's also a bit inflexible, especially if you imagine publishing a model that's been trained on some kind of data, but then you have a different kind of data. You wouldn't… it wouldn't be obvious how to, um,
represent your data such that you could use that same model, um, to work on it.
So, it's also interesting to consider more generic embeddings as, like, a potential alternative to this.
Um, and, uh, the most generic embedding that I know about would be something like text.
Where you can imagine sort of rendering all of your data in, like, a JSON or, like, a text-like format, such that it's, um,
totally, uh, trivial to combine data from many different formats into a single training corpus.
Which is why large language models were the first foundation models to be published, is because text is the easiest dataset to, um, combine.
Um, so, uh, there are a lot of challenges with this, like, more speculative, um, alternate to, uh, dedicated embeddings.
Um, the biggest, uh, obstacle to this is that language and text are, like, not, uh,
natively, uh,
They don't… they don't operate well, um, with each other, and language… language models especially, and sort of the tokenizers that they use to represent language,
don't really capture what makes numbers different from other types of language tokens.
And so, common tokenization methods for numbers,
um, are pretty rudimentary, in a sense, if you think about it. Like, you can sort of alternately…
tokenize each individual digit, but then that sort of takes a lot of the contextual information out of your representation.
Or you can imagine tokenizing every digit as its own token, which clearly does not scale if we try to tokenize every number. It's like a nonsense, uh…
task. So this is clearly different, uh, difficult, and, um,
One possible way of getting around this is by imagining creating dedicated numerical tokens to represent a continuous numerical spectrum. So, how do we make this continuous?
is by scaling the magnitude of the representation, so… such that a larger number would sort of be the same, like, number
object, but would be larger in embedding space, um, in magnitude.
And, uh, this worked better than we expected, um, especially in out-of-distribution generalization.
This is an interpolation plot that interpolates, um, so this whole region has not been seen during training.
And if we combine this method, or if we compare this method with other, sort of, standard numerical tokenizations,
Only this kind of continuous tokenization gets close at a smooth interpolation out of distribution.
And I was quite surprised by this, that the models, um, trained using this continuous tokenization were able to even reproduce, um,
planetary orbits, um, even though they're not trained autoregressively. And you can see that, uh, non…
continuous tokenizations just completely fail at this type of task.
So, um, who knows? It could be that, you know, maybe, uh, we…
will want to have an alternative to specialized tokenization schemes for all of our variety of datasets out there.
Um, regardless, I think, um, a message I want to send here is that I think language will be a critical
modality or input, um,
Whether it's used as a data format or not.
And the reason for this is basically context is going to be critical, especially if we imagine putting multiple datasets into a single model.
Like, these are two images, but they represent completely different scales of what they're capturing. Um, this is, of course, a black hole on the left, and then on the right is, like, an image in a calorimeter at Atlas of a pion.
So, um, even though these both are sort of, like, pixelated scientific images,
to a model, um, it's critical that the model understands what it is… what is being captured in each of these settings.
And, um, we see that this actually has, like, tangible performance outcomes, too. So, um, I like this example from… this is just time series forecasting.
Um, that these researchers found that context is key in the sense that by providing just a little bit of context to a time series prediction model that's been trained on a very diverse source of different time series,
Um, having just a little bit of context about, like, what, uh, the time series represents, when it was captured.
really constrains the overall uncertainty of the, um, rollout predictions of where the time series should go.
And I think it's likely that this will hold up in, um, in physics contexts, where our model is having to make predictions across a variety of huge scales.
And, um, we're also seeing that… Oh, yep.
So, you took a model…
It was just built to…
interpreted time series.
And you told it something about historical data, or…
power plants.
whatever. Sure, right, yeah. In this case, I think it's power plants. And it was able to take…
that information and get a more accurate prediction for protection.
That's right, and these blue bands are sort of representing… they test the model many times to sort of ask it to complete the time series.
And the variation is huge, um, unless you provide a little bit of context in which the variation gets… And that context was just that bit of text that you picked. That's right.
Wow. Yeah. I don't know, this…
This is a model trained on our, uh…
What is it? It's real… it's time series data, but it's across, like, a number of different, um, settings, so it's not just power plants, um… Oh, okay. The additional data is just those sentences.
I was telling it prior that it's, you know, it's a solar power, so of course it should…
go down during the nighttime, right?
definitely, and I think it's not obvious if… We have all sorts of time series, how it is…
Where… was his previous experience like?
power plants and five other different types of time series, or was it a million different?
other time touches, any kind of time series going up, I mean, what does it do? I think it would crane up the stock market, yeah, like, um, population, like…
traffic. Yeah, but it really makes a difference whether the…
The training data,
fell into a few discrete categories, and that sentence was enough to tell her, oh, it's…
my Category 5. Right, right. Or whether it was a million different…
Categories. Yeah. And then it was smart enough to…
somehow guess which… Right. So I don't know from what you've told us which of those things is true. I believe this one was trained on, um, the Monash time series benchmark dataset, which
Uh, off the top of my head, I think it's probably, like, 30 broad category of time series data.
But, um…
I think the settings are quite diverse, so I think just… I don't think, like, power plant…
time series isn't… is…
Um, one category, so, um…
And I think you're right that, you know, of course it has a little bit more information, so it… I don't think that this is super surprising. I think it's more…
It indicates to me that if we train a model in an unconstrained way on very diverse data sources,
It is going to be helpful to provide context as we prompt the model to make predictions.
So, yeah, I think that this gets more extreme as you have more diverse training data.
So, I actually wanted to ask you about your numbers, uh, work. So… Oh, yeah. Like, I don't know if… I'm sure everyone's noticed, but, like, the…
chatbots have gotten a lot better at math in the last few years.
So by now, you can almost basically count on them to do arithmetic and bookkeeping correctly.
That's right. So do you know what they've done under the hood? Have they used your work?
For the… They use tools. They use tools, yeah. It secret stuff they can't tell us about. At the moment.
Basically, they use Python, um, or they'll use a calculator. Um, so, uh, that's what's happening under the hood, is that the core representation
has not gotten much better at arithmetic, and I actually think I have a backup slide on this, um, which I thought was kind of fun.
And so that's sort of distinct from what you guys were proposing?
You're proposing a different representation?
Right. It's not just tokenizing all the numbers, but actually, you had a list of how you could represent the data, uh,
And one of your options was just tokenize all the numbers…
Straight up. Individually, right? Individually, and that's still what they're doing, probably?
Often what they do is they tokenize numbers, um, like, 3 digits at a time, in reverse order, because the, um, later digits are more important than the earlier ones.
So there's, like, some of these twists, but really, when you are interacting now with, like, ChatGPT or Claude, it's just calling Python under the hood. So, um…
We tried to really focus on settings where it's not clear, um, what function you would use to solve the problem. Like, there's no…
Um, there's no Python script to run to make that prediction, because we wanted it to run directly on data.
Um, and…
Yeah, you're right that these models have not been good at arithmetic, and this is sort of the classic, like,
Back in the day, when you would ask ChatGPT to do four-digit multiplication, it would fail over 90%, 95% of the time.
And I checked more recently, um, because, you know, DeepSeek and some of these other models have come out, and even this
$700 billion parameter model.
This is also multiplication tables, um,
it maybe memorizes up to, like,
10, or maybe, like, 12 by 12-digit accuracy.
Um, but then it drops off, so the language models are not understanding what does multiplication…
Meaning. True. They are equipped with now tool use.
Yes. Calculator use. Yeah.
The second one, it's not using a calculator? No. It doesn't understand, like, long multiplication or whatever, doesn't… It doesn't understand those rules, yeah.
You need to have a use thing, you know, based off the fact that it's bad, or, like…
I mean, obviously, like, I guess, how would you know whether or not it is using a cover?
Hmm. I mean, of course there's, um…
I guess these are from papers where they're able to use the open source model directly, and so the researchers know
how the model is being, uh, evaluated. Um, but you're right that in settings like
working with GPT now, we don't have perfect insight into what is happening under the hood as it's making those calculations.
They want to make you think that it can just calculate it, right? Right. Secretly, it's like…
Yeah. Pretty misleading.
Um…
Cool, so one more quick comment on language, and then I'll move on to the last part.
Oh, Professor? Professor Petey?
Yeah. Yes.
Hello, can you hear me? Hi, uh, can you go back… can you share your screen again? Um, because it looks like a…
Ah, thank you.
Yeah, sorry.
Yeah, of course.
Thank you.
There we go.
Perfect, alright, that works. Thank you!
Of course. Um…
Great, so last comment on language is that, um,
you know, maybe it's useful as a data format.
Maybe it's useful as…
context for a model as it makes predictions across diverse data.
Maybe it's also useful as an input itself, like, it's arguably, like, a pretty important
type of data that we use in physics. It embeds a lot of contextual information.
And so, um, there's been some interesting work
recently about, um, using
papers, um, so it could be… it could be, um, text in the form of, like, um,
captions on an image, but we don't have a lot of captioned images in physics. They're sort of hard to aggregate, and so, um,
more useful, maybe, is the sense of scientific papers, and so you can imagine
pairing scientific papers with actual spectra, or, like, scientific data sources.
And using a contrastive loss to, um, force these representations to live close together into a models embedding space.
And, um, this paper argues that this improves the quality of some of the downstream tasks that you might want to do on Spectra alone.
And so, um…
Point being that it… I think we're gonna think creatively about how text…
should be included into physics analyses in the future, and it could be that we'll just consider text to be a critical data source on its own.
Um,
Yep.
Do I have a question about this? What do you? Yeah. What do you do if the papers are wrong?
Um…
I…
I don't know if this paper addresses that explicitly, um…
I think, actually, they may still be useful.
simply as, um…
embedding relationships between objects.
you know, the idea that, like,
galaxies have spectra.
spectra have whatever parameters, um…
you could think of the model as, like, frantically trying to learn, like, a world model of how the universe operates, and the spectra are giving it one source of insight there.
And, um, at the same time, language is also providing its own world model for relationships between objects that we study, so…
I think even if the numerical claims of papers are wrong,
Um… I still think there might be utility in, um, just encoding those relationships in language form.
in a representation.
What if you… what if you… what if you're trying to answer some question about electromagnetism, and you train on papers that all are based on the ether?
Yeah, clearly there's a limit, yeah. Um, and if you're… if you're…
Um, claiming that something exists that is not represented in your spectra or in your other data, right? Like, maybe this claims that there's aliens, but we don't have any spectra showing evidence of aliens.
You'd hope that a good contrastive loss should really only pair, like, what are…
the shared semantic concepts that are present in both inputs.
Um, and so you'd hope that the model would largely ignore noise, um, whether that's noise in the spectra or whether that's noise in the papers.
Um, but it's a… it's a really cool point, and it's something I've been thinking about, is, like, in a basic contrastive loss setting,
Are there ways that we can sculpt a latent space to have
shared information and not shared information.
And maybe the not shared information is noise that you want to ignore. So I think that there's more room for, like, uh…
more curated ways of representat… representing your information, um, other than just putting everything together.
And hoping that the model learns what's useful and what's not.
Well, part of it could be it has the author list, right?
I learned that, you know, certain authors don't pay attention to their paper. It's true.
No, I think more likely, it's really just the… the…
Um, it's weird because we think of papers as, you know, producing…
overall scientific, um, results, and of course they do.
But I think to a model like this, it's paying attention more to just sentence structure and, like, what sentences talk about what objects.
And, um, which authors are associated with the useful papers, maybe.
Um, just for time, I'll mention some of these last few slides, and then I'd love to get into more conversation.
Um, so, uh, I've talked about the challenges of
putting datasets together.
And now I want to think about, as we scale this to even more extreme versions, um…
what new problems might emerge.
So I'm thinking about basically taking information from across an entire detector and using all of those inputs into a single model.
And even potentially across many detectors, and maybe even different disciplines of physics, say.
So, um, first just to define, um,
modality is…
kind of a fuzzy concept, but it's a specific way of perceiving the world, and it's often, um…
Compared to the human senses. So you can think of, like, vision as different from touch, which is different from smell.
Um, in the same way, um, in machine learning, we talk about how to ingest inputs of different formats and different sources.
And in standard machine learning contexts, this really just refers to, like, a handful of specific data types, and usually they're, like, images, text, speech.
Um, but in physics, uh, we have way more modalities, and um…
Why do we even care about modalities and having multimodality in a model?
Um,
It's because the benefits have seemed kind of, um, evident in recent years.
That, um, including multiple views or multiple modalities in a single model can improve performance.
This is kind of the classic paper that demonstrated this, which was a clip from OpenAI in 2021.
And they compare, um, results trained on ImageNet, in this case, the examples they give are of bananas.
Um, and they compare CLIP, which explicitly pairs images and text into a single representation.
versus ImageNet, which is a large-scale image model trained only on images and only on ImageNet in this case.
And you can see that they get the same performance when they evaluate in-domain on the same bananas, the same datasets. But as you get sort of further out of domain on, say, like, more abstract
ways of representing the concept of a banana to, like, things that, to me, don't even look like bananas down here.
This clip representation remains really strong, um, at, at, uh, identifying these objects as bananas.
Um, whereas the image model alone, uh, really drops off in performance. So the idea is, like, having multiple views with different formats.
create some more robust embedding or a representation of a concept.
And, uh, we've also seen this in science, too, so it's not just an industry. So, um, AstroClip, um, came out from, uh, my team a few years ago.
Um, pairing images with Spectra, in this case, instead of images and text.
into a shared embedding space, and, um, this allows for
querying of nearest neighbors, and so you could sort of search, like, given a galaxy, give me galaxies that look like this galaxy. You can search image to image, or you can search spectrum to spectrum, or you can also cross the modalities, say, give me a spectrum that corresponds to this image of a galaxy.
Um, and you can also do downstream tasks using this embedding as a starting point. So the, uh, aligned embedding space here,
gets better predictions of redshifts of these galaxies than an unaligned representation that's only been trained on images. So, point being that, you know, there seems to be some useful information that has been retained in this combined latent space that exceeds
the performance of an unaligned latent space.
No, uh, that's great for two modalities, but we have a lot of modalities in science, and… Okay, I actually want to understand that example better. So, um, so…
puts the X and the Y axis of the…
of those 2D plots?
Down here. It's Redshift, true Redshift, predicted redshift.
And so a perfect model would have, like, a line on Y equals X for perfect predictions.
And this is just showing the R-squared score of, like, how well it aligns, um, with a perfect frame rate. I'm just trying to predict the redshift.
Given what? In this case?
This one is starting from the shared embedding space that has knowledge of images and spectra. Yeah. Um, and then this is just going from images, which should not give a great representation of redshift.
So the images, uh, are you getting the redshirt from the shapes of the galaxies, or, like, they're photometric? It's like a photometric redshift, or what?
Um, for this one, I think it's not clear. I mean, I think basically we would expect these results to not be so strong.
Uh, because I… it's…
2D image, um, across a couple different bands.
Um, so there's not a lot of information in Redshift that's encoded there. And then, in the case of the Astro clip, are you feeding it both an image and a spectrum, or just still an image, but it's going to a better late space?
It's starting from the embedding space here. So, the embedding space is like a fixed dimensionality, um, but it has been trained to align
semantic information from Spectra and images.
However, how would it… sorry, how does it start? Does the embedding space have a ZHP
Didn't it? Or how does it start from the… Oh, well, it starts from this, like, I think it's 768-dimensional space.
And then we train another, like, lightweight neural network on top of that to predict the midshift.
Right, but where's the true richest come from?
Uh, well, we'll have labeled data, and so we evaluate it on some labeled test set.
But the label is not accessible to the model.
Right, yeah. So, but you must be feeding it either images or spectra or both, then, right, to get to the embedding space.
Like, you're not starting for the embedding space, you must be starting from…
an image or something. Otherwise, it's not a fair comparison with the…
Oh, sure, um…
So, uh…
the embedding spaces of a single galaxy.
It has, uh, access to both image and spectral representations, and so in this case, uh,
Right, yes. These plots are both sort of the image
portion of the embedding. There are other plots that show the spectrum-only, um, predictions, and they're much sharper, as you would expect, because the spectrum encodes much more information on the redshift.
So this is really kind of probing, like, the image portion of these intentions. So you can improve the image base, like, regression or prediction of a redshift?
By having this shared embedding. That's right, yeah.
And the… if I showed the Redshift version, you… the argument is kind of the opposite of, like, the shared embedding space.
is still able to give good predictions at the redshift, and it hasn't lost information.
Yeah, good question.
Um…
Uh, I think I probably just have, like,
2 minutes left or something? We ask a lot of questions. We started a little late, so, yeah. I'll say 5 minutes, yeah. They're really good questions, so I appreciate it.
Um, okay, so what I just showed was for two modalities.
But it's not obvious how to extend it to more modalities than just images and spectra, but in physics,
Our detectors capture a lot of information across many different, um, parts of the machine all at once.
So, arguably, you know, if you think about all the components that go into
a particle physics detector, or even a telescope. There are a lot of different modalities that you might argue are present.
Um, uh, s…
similar illustration of the same thing, you know, maybe you have tracks, maybe you have…
different layers of calorimeter images, point clouds, you might want to include all of this information into a single model.
How do we do that? Um…
Well, first of all, we've, um…
just produce some datasets to try to help enable these types of studies.
So this is a large-scale data set called the Multimodal Universe.
That, um, takes a lot of publicly available, um, astrophysics data and sort of makes it, like, machine learning ready. It sort of renders it in a format that's
easy to combine, has clear metadata, and is released on Hugging Face.
Um, and using this dataset, um,
Uh, we trained this foundation model called ION1,
Which is an astrophysics foundation model that's been trained on 39 different modalities across 5 different surveys.
So, there's a lot of cross-matching between the different surveys that are indicated by these lines here.
And the types of modalities are sort of illustrated in these little cartoons. Um, so we have things like spectra,
Um, galaxy images, um, metadata in the form of, uh, tabular information, etc.
And, um…
So in order to handle 39 different inputs with different formats, um, we trained a bunch of different tokenizers that were meant
to tokenize, uh, all of the different types of properties the model might encounter, whether that's images, spectra, or more.
It's all fed into this kind of, like,
masked modeling, uh, framework, so it's GPT-like.
But instead of predicting the next token, um,
It's masking random tokens, um, as part of the overall inputs.
And then this overall embedding is then used as a baseline for a number of different downstream tasks that were not part of the training process. So that includes regressing physical parameters, um,
us doing instant segmentation or image segmentation, so maybe you want to take a galaxy image and then train a lightweight convolutional neural network on top of it, and then that network will sort of identify, like, important parts of the galaxy image.
Um, or you can do retrieval, so maybe you start with one type of image, like a strong lens.
And then you can do a cosine similarity, and you can easily find other images across all of these different inputs that, um, are similar to that image.
So, um, this was a cool stab at sort of how to extend this to, like, on the order of tens of modalities at once.
Um, it's not clear if this is the only framework that is able to
taken, um, dozens of modalities.
Um, but I think this is a quickly developing space.
Um, and here… and another exciting consequence of this is not just, like, you know, here's what it can do on astrophysics data.
But, um, we're starting to see some signs that, uh, some of these models can also seemingly transfer out of domain, um, to new settings.
So this is an early example of this, um, fluid dynamics foundation model that was trained on incompressible Navier stokes.
And then evaluated, like, out of domain on compressible Nadia-Stokes simulations.
But in two different settings, where, um, there's the NEAR regime, where the data kind of looks more similar to the training
domain, and the FAR regime, where the data looks less similar.
And in each of these two settings, the pre-trained model on physics data outperformed training from scratch on these baselines.
As well as outperformed video models that had been trained on similar amounts of non-physics video data. And so there's, like, some early indications that, you know, training on physics data
is useful for, um, transferring out of domain on physics data.
And then more recently, um, we've seen a number of different foundation models, for instance, in particle physics, there are a handful, um,
And it seems like some of these
transfer surprisingly well out of domain also. So these are two evaluations of one particle physics foundation model.
on, um, cosmological data.
As well as, uh, molecular data. And so, basically, they use the particle physics model as the baseline, and then they do just a little bit of fine-tuning, and then they're able to use this model on
very different types of, uh, data sources.
And, uh, in such a way where the predictions, um, outperform training from scratch on these types of methods.
So, I think these types of questions, or sorry, these types of results
challenge what we think, um, in terms of training data set curation.
for large-scale models.
It seems like maybe at some threshold of model capacity.
Um, historical definitions of, like, where scientific domains begin and end might not be so important to a model, and what might be more important is maybe, like,
Is the data structurally similar to one another? And maybe the semantic content is not so useful.
But, you know, we're kind of just scratching the surface of, like, what makes training data useful.
Um, and what makes it easy to transfer to a new dataset out of domain.
Um, so to sort of help…
explore this territory, um, I've thought a bit about, like, mapping scientific research with knowledge graphs.
To kind of get a sense of, like, what concepts seem to be related in scientific papers.
And maybe this can help us define what types of datasets might be considered appropriate for in-domain and out-of-domain applications.
Um, and, uh, where I'm really interested right now is a question of, like, the information content of these datasets, and can we say anything about them?
Um, from the lens of information theory. Um…
So, uh, uh, in particular, thinking about, like, the mutual information between different inputs, and wondering, like, what is the right combination of data that is similar versus data that's different?
In order to make, like, a good recipe for a strong foundation model that can transfer very broadly across fields.
Um, so to help with these types of studies, um, a team of collaborators at Flatiron in Wisconsin
produced this, um,
series of datasets. Sorry, I'm gonna take a quick drink.
Basically, we produce a series of datasets that, um,
can resemble non-trivial physics datasets, but nevertheless have known mutual information.
And this could be true for two modalities, or it could be for dozens of modalities. And this is a real improvement on the current state, because right now it's hard to… it's very hard to estimate mutual information in real data.
And if you want to know the true mutual information in your datasets,
More often, you just have to stick with very simple data sets, like correlated Gaussians, if you want it to be, like, fully tractable.
Um, and, uh, datasets like ours allow for kind of non-trivial studies.
using information theory to understand the relationship between different data sets.
And it also allows us to benchmark mutual information estimators to kind of give you a sense of, like,
how hard it is for current mutual information estimators to actually get the right answer, um, where all of these dots refer to different, um, popular mutual information estimators.
the perfect estimation would be this red line. And there's a lot of variation, so, um…
Uh, it's very hard to know the ground truth, mutual information, and real data, but if we had access to that, it might really change how we think about representing our data in a model.
So I'm optimistic that stuff like this will help sketch that out.
Um…
So, in conclusion, basically, um,
thinking about, uh, what we've seen in other industry foundation models,
is that, um, a lot of their… a lot of the excitement around foundation models in industry is not just the fact that they are large or trained on diverse data.
But it's that, at a certain threshold,
They exhibited these emergent capacities.
that were not necessarily expected. Um, so this is as a function of model scale on the x-axis, and then the, um…
figure of merit of various, um,
various tasks, um…
And, uh, I think this really drove a lot of the excitement into investigating foundation models, is that it seemed like emergent properties were suddenly happening at certain scales, um, and also with a certain amount of
diverse data, um, and a certain amount of, uh, size of the overall data set.
Um, etc. So I think it's still an open question, like, will we see something like this in scientific foundation models? So…
If we imagine training larger models on very diverse physics datasets, should we also expect some kind of emergent properties?
Um, I don't know the answer to this, and part of my…
messaging, I think, for this talk, is that
I still think that this exercise of, um, creating representations of diverse physics data is still useful.
Even if this is… if this never happens, um…
I think just putting these different layers of the map in conversation will likely point out really interesting, um, patterns across scales in our data.
And, uh, we'll also just enable us to be
more playful and more curious with our… with our analyses.
So, um…
Normally, I like to talk about the title of my talk here, but I think, um…
I think maybe just for time, um, I can pause here and, uh, we can take any questions, and if anyone wants to know about the title of my talk, you can stay after and ask me how that?
Thanks for your time.
Hey, uh, thank you for the very nice talk, and we have time for questions.