Kushal Vajrala

Robot Crawler with Reinforcement Learning In-Progress Notes

Kushal Vajrala — Wed, 26 Aug 2026 16:40:38 GMT

After a long and difficult conversation with myself, I decided to go down the career path that would bring me the most satisfaction. Part of that is building in public and doing the things that would signal that I actually know what I'm talking about. It's too easy to outsource our thinking and ability to AI so I want to ensure that doesn't happen here.

Reinforcement learning and Robotics are two fields that have always fascinated me but I hadn't taken agency over my professional career to pursue a living in them. I no longer have to separate interests and my career!

The goal of this project is to teach a robot with legs how to walk. Maybe I can extend it to navigating an obstacle course but for now we'll leave it here.

The following is a list of notes that I am taking while I learn the necessary prerequisites. It's not meant to be comprehensive or helpful to anyone but myself, but I want to make this public to let people offer any feedback or anything else.

If you don't see much notes, that's fine, I'm probably just busy actually writing them and they'll be pushed soon.

RL Notes:

Robotics Notes:

The Watcher

Kushal Vajrala — Mon, 04 Aug 2025 04:56:22 GMT

“No you don’t understand, I’m being watched.”

The man I was sitting next to on the bench outside my favorite shawarma place shifted slightly in his seat, but he made no further interruptions. The river across from us murmured, curious as to our conversation. Joggers ran by. A singular boat bobbed up and down right across from the bench. My eyes caught the fraying of the cardboard sign he held, a testament to its age. “Will listen to your problems if you pay me.” Glancing back through his matted beard and hair, I felt a strange sense of security and curiosity emanating from his eyes.

“I’m telling you, this kid looks too much like me. I dug through old childhood photos and I swear it’s a doppelgänger.”

I motioned to pull my phone out, attempting to provide some semblance of proof for my ridiculous claims. The stranger waved his hand, dismissing it.

“I believe you. Carry on, why does he bother you?”

Relieved that finally someone didn’t doubt it, I took a deep breath in and proceeded in my story. “Do you mind?” I asked, pulling out my Marlboros. After a small shake of his head, I took a puff and snapped my lighter closed.

“I first noticed him on the anniversary of my ex and I a couple months ago. Then, I was more concerned than anything, a baby crawling around on the grass outside wherever I was. I thought I was just seeing things, but he seemed to age at an accelerated speed.”

I took a break for another drag. I offered a cig to the man, but he kindly and politely refused.

“So initially, I went out to try and get him to his parents. But every time I left the building I was in, he would disappear! Poof, vanish, like he wasn’t even there. As I walked back inside, though, I could feel him materialize. His eyes, dripping with innocent malevolence, locking back onto the back of my neck. Watching my every move. Staying outdoors or maintaining eye contact are the only ways I’m able to stay sane.”

I felt a phantom sensation tingle down my spine, and I paused to scratch my neck. I envied the crowds passing by. None of them had to care about anything other than their own problems. My eyes felt heavier than usual from the lack of sleep.

“He doesn’t let me sleep. I know he’s out there, staring, watching. Am I being punished? I don’t know.”

“Hm. What would you be punished for?”

I stubbed my cigarette against the bench’s arm without thinking, and I winced apologetically. He probably slept here, and doesn’t appreciate the bit of mess I’m making. I debate whether I let out my regrets or not. Fuck it. I’m already at this point.

“My ex and I actually broke up after she got pregnant.”

“Oh? Usually that’s cause for celebration, I thought.”

“Yes, but I wasn’t ready. I’m not sure how to explain it. She was more of a filler in my life. Just a vessel to keep me feeling loved while I looked for someone I actually loved. I couldn’t raise a kid with her.”

“Ouch, man. You never thought about someone treating you like this too?”

“At the time, no. I was on top of the world, just received a promotion, and I didn’t envision myself actually settling down with her. Someone to keep me confident, keep me feeling desirable.

“She wanted to keep it, but I refused to participate and kept encouraging her to get rid of it. I didn’t care that it’d been 6 years of us together. I was just focused on myself. I’d come home and start arguments almost daily, and the last few months of our relationship were often spent away from each other.”

I felt the shame creeping up in my cheeks, and the tears slightly welling up as well. Family and friends were supportive about me not wanting to have kids yet, but I knew that what I did was wrong. I didn’t have the right to cry about it now. I refused the tissue I was offered and sniffled a bit.

“But anyways, about a year ago, she miscarried. Probably from the stress of it all. And I’ve never heard from her since. Is this it? Is this what I’m being punished for?”

“Fuck man. You’ve been through shit and you’ve put others through shit, huh. Let me see a picture of the kid again.”

“I’ll do you one better. He’s a poster I made, back when I thought it was just a neighborhood nuisance playing pranks on me.”

I explained the strange situation: the kid was able to be photographed, but no one could see him but me. All of my family and close acquaintances thought it was a Snapchat filter or something of the sorts, just a part of my annual elaborate April Fool’s day prank. I gave up trying to convince them after a week.

“Anyways, the kid is growing older and looking more like me. It’s scary. I’m not sure I like it. Why me? What am I being punished for? I feel needles on me at all times. I’m exhausted from being vigilant all the time. I’ve resorted to dragging my bed to the window so I can watch him and fall asleep from the temporary reprieve in exhaustion I’m blessed with.

“I can’t exactly stay outdoors all the time or look at him. I’ve asked for work from home for my job, and I’ve found a cafe with outdoor seating even in this cold to work from. But I’m tired. I want to stay in the warm indoors without being obligated to look at this kid. I can’t eat or go to the bathroom without multi tasking. I can’t do anything. I hate this. I hate this so much.

“Is this because I threw my friend under the bus when we both got caught cheating in college? Is this because I took all the credit and got a promotion instead of my coworker who deserved it?”

I buried my head in my hands and continued confessing my sins, each one being pulled out like intensely dense pieces of mucus in my sinuses. Felt better while I talked about it, but still left a bit of a hole inside me.

I kept going. Then, I chose a moment to stop and look up. My fried nervous system couldn’t be bothered to get shocked. There he was, standing right in front of me. Eyes burnt black like coal. My cheekbones and ears. Nicole’s nose and smile. A man. I reached out a hand, and the apparition crumbled into dust in front of me.

I looked to my side and watched the man stand insouciantly, brushing the resulting black dust off his pants.

“Was that… my son? My child?”

“Yes. What could have been.”

“Ah. Then just a punishment, for my sin?”

“You’ve committed a great many. That was just the start.”

I also stood up, dusting off the remnants of the cigarette off his bench. He grunted an appreciative sound.

Without a word, I followed the man to his boat. He got in effortlessly, but held out a hand to stop me from getting in.

“Where’s my money?”

“Oh. Yes, of course. Thanks for listening.”

I almost held out the money, but a doubt stopped me.

“What’s your name, if I can ask? I’m Will.”

“Charon. Like ‘care’ and ‘on’ together.”

“I see. Why is it two dollars? I thought it was only one coin?”

“Even us deities are not free from inflation.”

I laughed and held out the money, which he gracefully accepted. Compared to the past few months, I suppose death is preferable to this.

I clambered into the boat, and we rowed off into the distance.

Hiatus

Kushal Vajrala — Mon, 04 Aug 2025 02:37:23 GMT

or, a dusting of my little corner of the internet, and what the future beholds.

A Recap

Hello. It's been a while since I've last written a post for my blog. It was, at this point, more than a year ago when I was so passionate about expanding my knowledge, working on new projects, and making a bit of a name for myself. Lots of things have happened since then. Let's do a quick recap.

I used to not have a chair for my desk. I found that standing helped the creative juices flow better. It wasn't until I realized I stood for 10 hours in a row trying to fix a problem with my custom ChessGPT that I realized my feet did not appreciate the sacrifice on their part to let me continue to work and write at the pace I wanted.
A few short stories have been written since then, the one I am most proud of saved on my notes app on my phone. Perhaps I'll share here one day.
Many friends made, some friends lost, and existing connections grew stronger.
A promotion at work! Lots more responsibility and lots more stakes assigned to me. I've learned a lot.
I got my wisdom teeth out! The procedure kicked my ass, but I'm glad to have recovered from it.
So so much more. If you'd like to know more or want to chat, please, shoot me a text or email and I'll reach back out to you.

The Future

I was urged by a friend (whom I started writing blogs with, you should definitely check him out. Be warned, some of it is a bit dense if you are not a Physics graduate student) to start writing blogs again. Having accountability is usually the most efficient when it is intrinsic, of course, but I greatly appreciate the external source, especially when it comes from a friend.

I have come to the realization that I spend far too much time consuming and not enough time creating. I am grateful and privileged enough to live in a time period where I have both the technological means and the living conditions that allow me the opportunity to create. It seems a waste to not contribute whatever insights I have, original or not, back to the world through my corner of the internet. At the very least, if I am only helping my future self (re-setting up this blog post again was pretty easy because of what I had written earlier), I will be satisfied.

I want to learn. I want to grow. And, finally, I want to teach and pass on what I have learned to anyone else. I might incorporate my sports commentary or creative writing on here as well. That's okay too. I just want to get back into putting the pen to the paper (or rather, my fingertips to the keys) at a regular basis so that I am able to leave something that will be of use to anyone.

Thanks for reading. I will be writing much more. Till next time.

Kanji to Hiragana

Kushal Vajrala — Thu, 25 Jul 2024 02:37:23 GMT

or, a first timer's journey messing with chrome extensions.

Item Summary

With this, the 日本語 learner can switch between Kanji and Hiragana seamlessly, allowing for an uninterrupted reading experience.

Item Description

When reading Japanese as a new learner, the amount of new, unknown Kanji can become overwhelming, and can make people feel frustrated with having to go away from the screen to look up words. With this extension, learners can easily switch between Kanji and Hiragana to ensure an uninterrupted reading experience. They can also toggle specific words to switch back and forth for Hiragana/Kanji. They can also simply press the button again to switch back between the two.

Usage

Let's take a page from the Aozora Bunko, for example. You can notice all of the various kanji on the screen that a new learner might not know:

Now, clicking on the extension will give us an option to switch between kanji and hiragana:

After clicking on the extension, we can see that all of the kanji have been transformed to hiragana, and the kanji that have been transformed are underlined so the user knows which hiragana belong to which word:

Let's click back on the very first word, to see what its original kanji was! You'll be able to click on this multiple times to switch back and forth:

Awesome! We can see that 蜘蛛 is くも in hiragana! Using some reading in context, maybe you could have found out that it means spider. Now, what if we want to change it back? Clicking the extension resets everything to how it was:

Motivation

I didn't like having my reading experience interrupted, so I built this chrome extension. I hope you like the extension! Let me know your thoughts!

Cart Pole

Kushal Vajrala — Wed, 24 Jul 2024 20:35:01 GMT

or, how my love for video games and AI/ML brought me to an intersection of them with reinforcement learning.

The information in this post is not intended to be a thorough mathematical primer on the topic. Rather, it is simply to showcase and explain a cool concept I've learned and implemented myself, and to explain it to a layman. I will be discussing some math here and there, but I recommend the Reinforcement Learning book by Sutton and Barto if you're really looking to get your hands dirty.

Introduction

A common prank my friends and I like to conduct is to bring people to a game of Mao. Essentially, the uninitiated are not allowed to know the rules of this card game. They must figure it out by watching other people, and they usually end up suffering in-game penalties frequently as a result.

How would you approach this game (or, for the initiated, how did you first approach this game)? I bet that when starting the game, you would do a random move and see where that gets you. You would observe when you get penalized for something, and try to associate that pattern so that you try not to let it happen again. You might guess wrong, and then try a slightly tweaked version of your move to see if it happens again. Similarly with video games, I do the same thing: see what works, see what doesn't, and try to keep doing what works and what doesn't work. Lord knows how many times that I, someone who's never played DnD before, tried to adjust my Baldur's Gate 3 classes, actions, and story paths to get the most satisfying one across multiple playthroughs.

Definition

This, my friends, is the essence of reinforcement learning: how an agent conducts actions in an environment to maximize its overall reward. We'll get into what these terms mean in a second.

A very common image that gets utilized when describing RL is as follows:

If you recognize what shape this takes, you are correct in assuming that this is a Markov decision process. If you don't, no worries.

Let's break this picture down piece by piece. You have an agent (a model, a piece of code, etc.) that takes an action and interacts with its environment. Then, the agent is given two pieces of input from the interpreter (a fancy way of saying something that translates the outcomes of the action to the environment). The state, which you can think of a snapshot of the environment after the action, and the reward, which is an incentive to the model. For an RL algo trading stocks, the reward in simple terms could be its profits. For checkers, the number of pieces captured. In the case of Mao, the reward would be... well. Wouldn't you like to know? We then loop this process, and keep learning.

Note: the results of the prior action does not impact the probability of choosing the current action. Just a bit of a disclaimer before we move forward.

By this point, I'm sure you've been understanding everything I've said; however, one question probably still remains. "How?" And that, my friends, is what the next section is for. We'll discuss an algorithm that I implemented!

Q-Learning

Lots of people I know feel like this whenever they encounter/encountered a math test:

The Algo

Don't worry! I will do my best to walk you through the math. (If you weren't worrying at all, then that's good) Here's the equation for the algorithm we're going to cover today, Q-learning:

$$Q(S_t, A_t) \leftarrow Q(S_t, A_t) + \alpha \left[ R_{t+1} + \gamma \max_a Q(S_{t+1}, a) - Q(S_t, A_t) \right]$$

You see the $Q(S_t, A_t)$. The $Q$ is a function that we call a Q table. This is defined as follows: $$Q(s, a) = \mathbb{E}[R_t \mid S_t = s, A_t = a]$$

where the inputs to the Q table are the state s and action a that the agent takes. The $\mathbb{E}$ refers to the expected value, $R_t$ refers to the total reward up to the timestep (basically, what point of time we are in) t, and $S_t$ and $A_t$ are the state and action at timestep t. What does this mean? Basically: the Q table is keeping track of the cumulative reward up to time t for every (state, action) pair. Think about it: it's basically what you do in a game you don't know, right? You try something (action) in a certain situation (state) and see what that does for you (reward/no reward/sometimes negative reward). The Q table is a convenient way of keeping track of this.

Now, you'll probably have seen already that the original Q learning algorithm updates itself. That's cool, right? That's how this model learns. Now that you have an understanding of the Q table, let's go into the rest of the equation.

As we already covered, $Q(S_t, A_t)$ is basically the current estimate of the reward for the (state, action) pair at time t. What about the $\left[R_{t+1} + \gamma \max_a Q(S_{t+1}, a) - Q(S_t, A_t) \right]$? $R_{t+1}$ is essentially the reward after the action was done at timestep t, and the $\gamma$ is the discount factor (we will get into this in a bit). The $\max_a Q(S_{t+1}, a)$ is giving us the maximum possible reward for the next state at timestep t+1 after the corresponding action was done at time t; I know that was a mouthful, but basically, we're figuring out which action a at timestep t gives us the best reward for the next state, and we're rolling with that. The sum of these two terms is the target value. Our reward at this spot, plus the discounted future reward given that we take that action.

If you've taken a finance class before, you've probably learned about the time value of money. Essentially, $100 today is worth more than $100 next year. Why? The simple answer is opportunity cost. However, if you aren't satisfied with that answer, or desire a more technical proof/explanation, feel free to look for one. We apply this opportunity cost with $\gamma$, our discount factor. We discount future rewards, since the present rewards are worth much more, hence the multiplication. As you can guess from the explanation, the discount factor is usually between 0 and 1, though closer to 1. We don't want to discount it TOO much.

Subtracting the original $Q(S_t, A_t)$ gives us something called the *temporal difference (TD) error. Basically, we're seeing how far away our current Q value in the table is to the target value (which is the sum of the reward and the discounted max possible future reward that we discussed earlier).

We then multiply this whole thing by the $\alpha$, the learning rate. We don't want to immediately swap out our Q table values to this one! We want to gradually make our way there, because we have no idea if what we've learned is exactly true or not. Similar to our Mao example: you don't know if the reason you didn't get penalized is completely due to you guessing the right rule. You might have gotten lucky! So you remember it, and keep an eye out for it next time, adjusting your expectations accordingly. You've by now deduced that the learning rate is a value between 0 and 1, but closer to 0, since you don't want to update your values too quickly.

Now, for the final part. You adjust the $Q(S_t, A_t)$ by adding what was in there previously with the learning rate times the TD error! This gives our Q value a little nudge in the right direction.

Implementation Explanation

But "wait!", you might say. How do we even start? Looking at the diagram earlier, it looked like a never ending loop. When do we start, and when do we stop? How do we make Q values in the first place? Let's get into that too.

Let's answer the easier questions first.

When we start this, we initialize our current state to 0s (or nulls, depdending on how you would like to implement this), and we initialize our Q table to have all 0s as well.
Starting and stopping involves two things: number of episodes, and the number of time steps per episode. What's happening here is that each episode is the number of times we let the agent explore the environment (or, more simply, play the game) and the number of time steps is essentially the total amount of time we give the agent per episode.

Sure, that's all cool. But what happens if the Q table contains all zeroes? How do we even start populating it with values? That's a good question. We're going to get into something called an epsilon ($\epsilon$) greedy strategy.

$\epsilon$ Greedy Strategy

Let's go back to the initial discussion of the game of Mao. When you start playing the game, you don't know anything except what your options are, which are to essentially put a card from your hand down. That's all you know. From there, you randomly experiment and try to see what works and what doesn't work. After you've gotten a hang of the game, however, you don't want to keep doing random stuff: you want to go ahead and try to use what you know to win. This is known as exploration/exploitation.

When our agent first starts interacting with the environment, let's give it some freedom to just randomly explore and try things and see what that gives it. However, as the agent gets a hang of things as it plays through a good number of episodes, we want it to exploit what it has learned in its Q table for the max possible reward.

This can be done through any sort of setup. It's really up to you regarding how you wish to set this up, as long as it achieves the goal of decreasing your $\epsilon$ (exploration/exploitation coefficient) over the episodes so the agent can first explore, then exploit. Here is my implementation: $$\epsilon \leftarrow \epsilon_{\text{min}} + (\epsilon_{\text{max}} - \epsilon_{\text{min}}) \cdot e^{(-d\cdot p)}$$

Where p is the episode number, $\epsilon$ and $\epsilon_{\text{max}}$ start off initialized as 1, and $\epsilon_{\text{min}}$ and d ($\epsilon$ decay: the rate at which the $\epsilon$ value decreases) are usually initialized to values that are between 0 and 1, but usually much closer to 0. You can probably intuit that as the episodes go on, the $\epsilon$ value decreases. But how do we use this? Simple, really.

Every time step, we pick a random number between 0 and 1. If this number is greater than $\epsilon$, we just use whatever is in our Q table and make the decision from what we've seen. However, if it's less than or equal to our $\epsilon$ value, then we just randomly pick an action and roll with it, and see what that gives us.

Once again, you'll be able to intuit that the first case probably happens later on in the episodes, while the second case happens towards the beginning of the episodes. We're done with the theory!

Mess w/My Code!

You might not have a perfect grasp on the algorithm yet, especially if you haven't recently been studying or doing math. That's okay. Seeing it in code form will help you out there.

We're going to be using Gymnasium to help us set up. We're going to be using their Cart-Pole game.

What's happening here, and why did I choose this environment to do Q learning in?

Cart-pole is a famous intro for those who are dipping their feet into due to its simplicity. Basically, you are given a cart with a pendulum on top of it. The cart rolls on a frictionless surface. The action space (what all can the agent possibly do here?) is simple: push the cart left, and push the cart right. The objective is simple: keep the pendulum up for as long as possible. It's considered a "fail" if the pendulum falls below a certain angle.

Here's my GitHub repo for my Q learning implementation of cart-pole. Read through the file, see if you can understand it and try to rewrite it yourself. I've included some comments explaining stuff, as well as links to the documentation if you want to understand it more. Don't hesitate to leave a comment if there's anything you're confused about.

Installation/Prereqs

Last blog post was Hugo. This one is Python. Ensure that you have Python and pip installed, and then run the following commands. Note, this is assuming that you're not using conda.

pip install gymnasium
pip install numpy
pip install matplotlib
git clone https://github.com/vajralakushal/cart-pole.git

Now, cd into the directory of the cart-pole and run:

python cart-pole-q.py

I've given you the option to select some paramters. Once you do, the model takes some time to train, and then a pyplot window should show up, plotting the average reward of the agent across every thousand episodes. Can you come up with a set of parameters where the cart-pole survives the longest (i.e. where the graph has the highest points).

Sources

Huge shoutout to the videos in this playlist. Heavily inspired me to write my own intro to this topic. Everything else is linked accordingly.

Thanks!

Thanks for reading! Feel free to leave any comments if you have any!

Pilot

Kushal Vajrala — Thu, 18 Jul 2024 03:48:20 GMT

or, a first timer's journey messing with static sites.

What is Hugo?

I'll just link them for you and save you the trouble of having to read my own paraphrasing. I can't resist, however: it's essentially an open source service that helps you generate static websites. Documentation, blogs, portfolios, you name it. I was inspired by Zoe's blog and wanted to create my own.

The Goal

This blog post is essentially a letter to my past self. If I had this blog post maybe a few hours earlier, I could've saved myself a lot of headache.

Installation

Currently, I'm working out of my Windows PC, so I'll go over this part assuming you are doing the same. If not, refer to this site.

[ ] Ensure that Chocolatey, a windows package manager (similar to brew on MacOS or pacman on Arch. I do not use Arch btw). Same thing with Winget
[ ] Funnily enough, same thing with Powershell 7
[ ] Run this command, but in Powershell 7: choco install hugo-extended

Site Creation

Yay! Everything has been installed and is ready to go. I use VS code just because the files are not as numerous to where I would need to use neovim or anything, but use whatever you're comfortable with. Just make sure to use Powershell 7 as your shell here on out.

Check to see if Hugo was installed correctly with hugo version. At the time of this post, it should be around 0.129.0.

Here comes the tricky part. To be fair, it was only tricky for me simply because I already had the makings of a barebones Next.js website, but I was tired of the learning curve. I'll definitely come back to it one day, but Hugo just provided an easier option.

Create a repository on GitHub and name it "username.github.io". (Does the period go inside the quotations or outside? It looks better inside but logically it feels better outside). Clone into this repo. Now, here's where the fun begins.

Navigate into the repo you just cloned and run this line:

hugo new site example.com

Now, we'll do a bit of configuring here.

mv \example.com\* .
mv \example.com\.* # should be redundant, but eh why not
rmdir \example.com\

If you look at your directory now, it should look like this

Great! Now, we can set up your hugo.toml as you wish. Add a title param in your hugo.toml file (anywhere inside works, as long as it's not part of a param):

title = Kushal Vajrala

Explore the different configs you can use. From here on out, it's just a matter of finding out your favorite theme: install and run for it!

The very adventurous will probably stop reading here, and go tinker on their own now that things are set up somewhat. However, I'm not totally sure that when I was working on this earlier. Let's use the theme that I'm using for this tutorial, hugo-coder. I was able to set up my stuff pretty fast with this clean, minimalist theme.

I would really rather not rewrite the download instructions they've so kindly given, so make sure you read and implement carefully. I recommend installing it as a git submodule.

Showing off your literary masterpieces

Now, here's where the fun (really, the writing) begins!

Assuming that you followed their example config setup, let's go into making posts! Your content directory should have a posts directory within it. If it doesn't, go create that.

Ensure that in your hugo.toml file, you have this:

[[menu.main]]
  name = "Blog"
  weight = 1
  url  = "posts/"
[[menu.main]]
  name = "About"
  weight = 2
  url  = "/about/"

I changed mine around a bit, because I just wanted one singular about page. For now, run the following lines:

hugo new \content\posts\hello-world.md
hugo new \content\about.md

You'll see that the files already come populated with some stuff in between some +++. These are essentially metadata that the theme utilizes to showcase and tag your posts. The prepopulated stuff can be found in the archetypes folder. Kudos to you if you explored a bit and found that out. It uses some regex to properly capitalize and figure out what to title your article. Fill the created files out to your desire.

Now, run

hugo server

and navigate to the localhost link that it gives you! Ta-da! Your website it almost done! If you want to deploy it with GitHub pages like I have, go through the following steps:

Push all of your changes and note which branch you used
hugo version to see which version of hugo was installed
Follow the instructions here
Navigate to your website, and there you go! It's all done.

Final Thoughts

I hope this helped you out! If you have any questions, feel free to reach out to me!