I'd like to persuade you why it makes sense to investigate deep learning models.
In our case, I'll talk about unsupervised segmentation.
As a short intro, my name is Piotr Migdal, and I have PhD in quantum physics and I work with Bartłomiej Olechno at ECC Games.
We use deep learning to create levels and physics engine for a race game.
Additionally we develop livelossplot, a Python package for charts for deep learning models.
And I also develop Quantum Game, in which you are able to play with quantum mechanics in the browser.
So before I talk about investigating models interactively, let's make some overview of general, how to track our developing model training process.
So the simplest way is to use, well console, and look what's the log loss, what's accuracy, et cetera.
But I don't think it's a good solution, it's very easy to miss that it gets better or worse.
So what the other step is to use TensorBoard or something which tracks metrics.
But I think it's much, much better to say, not only metrics, but also all parameters and code snapshots.
For example, Neptune, Weights & Biases or MLflow.
But it's not the only axis with respect to which we can look at model training inspection.
The other is depth.
So for example, you can only look at metrics, but I don't think it's enough.
Sometimes you can miss that it classifies, or misclassifies some obvious examples.
So we can always, and these are few examples to see what's going on, but I think it's much, much better to make interactive, and in a few slides I'm going to show you how.
When it comes to our project, we basically have high resolution scans of race tracks, we're going to turn them in semantic maps.
there is not enough data to supervised image segmentation, they're very diverse, but you can use something something like word2vec to look at adjacent locations, and it's enough.
But we want to make sure that we are able to extract relevant data.
It's a simple example of the scan but also @@@ of future map, not of the original pixels.
The question is if we can extract, for example, tire tracks or cracks on the road or in the pic.
And let me, and we then talk about that pipeline well here containing PyTorch, and in this case plus Ignite, we save models with livelossplot plus Neptune and metrics but also we, of the future maps, we create some visualization using D3, and as you see right now, I present them in a slide, in this case it's ioslides, when you write things, even for internal notes we use Distill for RMarkdown.
And here's an example, this is a picture, when we mouse over we see what are the relevant pixels or dots.
And not only we can look at similar pictures we expect in one place, but also we can look at the differences, for example, this one versus the other.
There is a river, there are some buildings, the river and there is forest, in this case we can see what's going on.
Also we can look at the places which do not work correctly.
But also we can use, well, game data.
In this case, it's SimCity, the classic one, the original SimCity, there's some buildings, and in some places we can separate residential area from industrial area.
Or we can look at another game like Minesweeper and compare models.
So you see, it's like we want to have something that tells us which regions is inside and which is outside.
This model shows something, but I think it's not perfect.
And this one is actually much better 'cause at the same time, you can separate inside from the outside, also see their different and here they're different.
Maybe you can look at the difference, yes, it's very remarkable.
And I think it's not that hard to include that.
In my model basically there are a few lines, the only thing is to write a simple visualization in D3, and also include JSON, so we need to export things to JSON.
But I think JSON is not that good for serializing matrices and arrays: it's costly, it's slow, it's not type-safe.
So I think it would be nice to have and use a general standardized format for input and output of matrices.
It's crucial, not only for deep learning, but also for quantum computing, it's basically the same story, just you need to have complex numbers to describe all of the data.
So, well, that's all, there's inspirations and also, especially, I invite you to look at the collaborative list machine learning, deep learning and statistics websites, a lot of extra explanations.
And, well, you can also check our personal project, this livelossplot and the Quantum Game.
Have a nice day.
