I recently decided to brush up on deep learning fundamentals, as I’m now deploying open-source transformer models for a webapp I made. (Shameless plug: it does text-to-speech for class readings… I made it for my partner who’s in grad school. You can see it here).

With that in mind, rather than (for example) go through the Karpathy videos or other existing curriculum, I decided to try to get Claude to build out a curriculum based on my specific knowledge gaps. I am iteratively building it as I go based on a rough skeleton I generated up front. My goal is to be able to reproduce the text-to-speech generation model I’m using, Pocket-TTS. Pocket-TTS does use language modelling, so… we’re going there.

https://github.com/cmoscardi/ai-for-ai – the repo where I’m putting materials. So far I have 2 “lesson” modules live. I use one of the reMarkable paper tablets so there’s a bit of infrastructure to get the generated notes onto that.

Admittedly, the title of this post is a little facetious. I have reasonable amount of background in mathematics and machine learning, including researching, developing, training, and putting into production “shallow” ML models. So I know my way around a loss function. That said, I barely remember concepts that are fundamental to deep learning, like the chain rule. I probably derived the chain rule in high school calculus, but at my advanced age of 34, my main recollection of that class is that my teacher loved the opera. (The things you remember). I wanted to really try to restart from the basics, in order to close some of the gaps in my knowledge that have opened up over the years.

What I’m hoping to learn

Beyond the thing itself (reproducing a non-trivial model), I’m curious how well this style of learning actually works. I’m no stranger to teaching myself mathy stuff… but will it be easier to get a grasp on these concepts by working through AI, with AI, and in AI? (shout-out to everyone who’s been to Catholic mass that I just triggered)

In seriousness, one of my least favorite parts of learning math from a textbook is that I’ll just get stuck on some logical leap that makes no sense to me, but evidently wasn’t worth another sentence or two to explicate. I’m hoping that I can actually go back-and-forth to get more meaningful clarification about that logical leap that I don’t understand, basically for free (or I guess however many trees each of my prompts burns down).

In addition to that sort of gap-closing, which I could get from the combination of an existing textbook and a chatGPT browser window, being able to specifically tailor the curriculum to my needs seems cool and useful.

What I’ve noticed so far

  • So far, what I hoped for has been true. What’s felt really nice about this style of learning is that I can specifically go back and ask, “can you flesh out XYZ part of the curriculum, I don’t understand this part”. With a couple back-and-forth prompts I can reliably fill the gap.
  • The first couple units (chain rule, write your own language model) are basically Karpathy videos. I asked for math, and it’s given me a bit more of the math and derivation than those videos do. I actually really like this, as I can somewhat reliably twist the curriculum to fill in my knowledge gaps. In other words, so far, tailoring works.
  • I am glad to have asked for exercises and labs up-front. I know I need that sort of thing to actually internalize this sort of material. That said, whenever I ask for help, or a clarification, on an exercise, Claude tends to want to just answer the exercise for me. An additional “don’t actually answer the exercise, just tell me xyz” in the prompt typically solves this.
  • I am curious where this is going to go as I get into the more arcane parts of what I’m looking for wrt. text-to-speech modelling. I suspect that SOTA TTS methods are pedagogically less well-travelled territory, so it will be interesting to see how clear those explanations are.
  • So far the scope of each lesson seems fine. I got through the first one in a single sitting. This next one will probably take 2 sittings (at a couple hours per sitting).

It feels weird asking an LLM for a curriculum about how to, essentially, create an LLM, but these are the times we live in I guess.