Aman Madaan

The case for inference-time compute

Fri, 22 Dec 2023 00:00:00 +0000

The next generation of AI systems for reasoning will likely rely on a powerful base language model at its core, with a number of inference time techniques that make it more useful.

Next token prediction is awesome

I’m in the next token-prediction is awesome camp. I think despite whatever the naysayers say, the ability to coherently complete a series of tokens grounded in some context is highly non-trivial, and I think humans do it all the time as well. While chatgpt helped in making debates around this stuff mainstream, it has been going around for a while now. Here is a quote from .

Next token prediction is not enough

As awesome as it is, I think the ability to do good next-token prediction is only a necessary condition to build systems that tackle difficult reasoning problems. It is also true that humans don’t just do next-token prediction for all but the most mundane tasks. Even if we focus on tasks that involve a single human generating some textual output, it is clear that the next token prediction exercise goes beyond just generation. For more challenging, creative tasks, we do next-token prediction + X. Here, X is a number of things, including:

Generate many candidates: We generate a lot of candidates and then pick the best one. Example, when suggesting a movie to watch that fits a certain criteria.
Iterate: Generate some output, then think about it, then generate some more output. Example, when writing a blog post.
Go beyond context: We don’t just use the context, but use the broader world knowledge to generate the output. Often, this involves learnings from the past (a memory of mistakes or cultural conditioning). Example, answering a question within our domain of expertise.
Use tools: we realize when we reach a point where doing things manually doesn’t make sense. At those points, we invoke a tool, and use the output from the tool to continue with the task. Example, we use spell checkers, calculators, etc.
Rephrase the question: Sometimes when we are stuck, we rephrase the question to make it easier to answer. Example, when we are stuck on a math problem, we try to rephrase it in terms of a problem we know how to solve.
Solve the easy bits first: We solve the easy bits first, and then focus on the hard bits. Example, solving a programming puzzle is often best done by coming up with a brute force solution first, and then optimizing it.

The examples above make a couple of things pretty clear:

A. Next-token prediction is not the only thing we do. B Next-token prediction machinery better be good, because it is the base on which all of the above is built. If the base is not good, then the rest of the stuff is not going to be good either.

There are now LLM techniques for realizing all of the above techniques, and are more commonly called search, memory, retrieval and alignment (going beyond context), tool-usage, prompting, and question decomposition. I think these techniques are going to be the next frontier of research in language models, and will be the key to building systems that can do reasoning. Together, these techniques require “throwing more compute” at the problem at inference time, using a good base model.

Distribution Transformation Perspective

My favorite way of looking at this is in terms of a distribution transformation. The base LM gives you a strong but simple distribution over the text on the internet. This is powerful, because you already have a great prior on to complete the text for a wide range of contexts. Let’s say this is the distribution \(P_{lm}(x)\). Now we can draw samples from this distribution, but it is likely not going to be very useful, as anyone who has played with models that have not been instruction-tuned will tell you. It is more interesting to parameterize a different distribution \(P_{lm + X}(x)\), where \(X\) is some transformation that we apply to the base distribution.

Acknowledgements: Thanks to Omar Khattab for the helpful brainstorming meetings. We were planning to write a paper on this back in December 2023 (after meeting at Neurips), but couldn’t find the time. I believe DSPy represents one approach to creating the transformations discussed in this post.

LLMs Are Stochastic Compilers

Sat, 01 Apr 2023 00:00:00 +0000

Or How to Think About Prompting with an Imprecise but Hopefully Helpful Analogy

Originally presented as a tutorial at the CMU-LTI Seminar

TLDR: Large Language models can be thought of as compilers, with prompts being programs written in a high-level language. Like programs, prompts can be written in different styles, such as by specifying instructions or a few examples or using alternative formats like code. Language models are stochastic, and programs (i.e., prompts) may require trial and error to produce the desired output.

Or How to Think About Prompting with an Imprecise but Hopefully Helpful Analogy
The Evolution of Abstractions in Programming
Common thread
- You are not teaching the task to the processor–you are just specifying the task differently. The processor already knows how to add two numbers.
Language Models Are Compilers
- Fleshing out the analogy
The Expressiveness of Prompting
- Specification with instructions
LLMs are Stochastic Compilers
Different prompting (programming) styles
- Let’s go back to our example of math reasoning
Summary and Key Takeaways
Acknowledgements

The Evolution of Abstractions in Programming

Here is a simple example of how abstractions evolve. Consider the problem of adding two numbers. Given a specification like “Given two numbers, return their sum,” we can write a program that solves this problem at different levels of abstraction.

Machine code: The lowest level of abstraction. The program is a sequence of instructions that are executed by the CPU. The instructions are represented as a sequence of bytes.
```
00101010 00000000 00000001
00101010 00000001 00000010
10010001 00000010 00000000
00111101 00000000 00000011
00111010 00000000 00000011
```
Assembly code: A higher level of abstraction. The program is a sequence of instructions that are executed by the CPU. The instructions are represented as a sequence of mnemonics.

MOV AX, 1     ; Load the first number (1) into register AX
MOV BX, 2     ; Load the second number (2) into register BX
ADD AX, BX    ; Add the numbers in AX and BX, store the result in AX
MOV CX, AX; Move the result to register CX
; The following code is platform-dependent and prints the value in CX

C and Python code: A higher level of abstraction. The program is a sequence of instructions that are executed by the CPU. The instructions are represented as a sequence of keywords.

#include <stdio.h>

int main() {
    int a = 1;
    int b = 2;
    int sum = a + b;
    printf("The sum is: %d\n", sum);
    return 0;
}

a = 1
b = 2
sum = a + b
print(f"The sum is: {sum}")

Common thread

There is a task we want to solve, with some input and output.
- Task: Add two numbers
- Input: Two numbers
- Output: Their sum
Each level of abstraction is a different representation of the same task.
- Machine code: A sequence of bytes, no translation required
- Assembly code: A sequence of mnemonics, assembler
- C code: A sequence of keywords compiler
- Python code: A sequence of keywords, interpreter
Yet, there is a common thread:
- A program is a way to communicate the task to the processor.
- Just different ways to represent the same task.

You are not teaching the task to the processor–you are just specifying the task differently. The processor already knows how to add two numbers.

Language Models Are Compilers

A useful way to think about prompting is as another programming language.
The language model is the compiler:

Input:
```
Task: Add two numbers.
Input: 4, 6
```
Output:
```
Output: The output is 10.
```
You wrote code in natural language, which the language model compiles into a sequence of instructions a processor can execute.:
```
Task: Add two numbers.
Input: 4, 6
Output: The output is
```

Fleshing out the analogy

If language models are compilers, what’s the processor?
- The GPU! The language model converts natural language into a sequence of matrix operations and non-linearities that the GPU can execute.
- If you are not convinced, you can perhaps think about FPGAs.
  - FPGAs are programmable processors that can be programmed to execute arbitrary logic. When you write code for an FPGA, you’re essentially changing the connections between its logic gates, thus dictating the flow of information through the processor. The FPGA is the processor and you write code to change the connections between the logic gates.
  - Similarly, the “prompts” you write affect the activations in the language model, and similarly change the flow of information through the language model. The language model is the processor and you write code to change the connections between the neurons.
- If you are still not convinced, I told you the analogy is imprecise.

The Expressiveness of Prompting

Recall: the goal is to specify the task, not to teach the task.

Specification with instructions

Generate a poem that rhymes with 'cat' and has 4 lines."

Bat, sat, chat, fat
A furry friend that's really neat,
It purrs and meows with its tiny feet,
It's the cutest thing when it's on your mat.

"Write a program in Python to test if `n` is prime"

#Input an integer
n = int(input('Enter an integer: '))

#If n is less than 2, it is not prime
if n < 2:
    print(n, 'is not a prime number')

#If n is greater than 2, check if it is prime
else:
    is_prime = True
    
    for i in range(2, n):
        if n % i == 0:
            is_prime = False
            break
            
    if is_prime:
        print(n, 'is a prime number')
    else:
        print(n, 'is not a prime number')

Specification with examples

AKA autocomplete: In addition to or instead of instructions, we can provide examples

Prompt:

    input_1

    output_1

    input_2

    output_2

    input_3

    ???

Feed the prompt to the model:

    output_3

Example: math reasoning

Prompt

Q: Jason had 20 lollipops. He gave Denny some lollipops. 
    Now Jason has 12 lollipops. How many lollipops did Jason
    give to Denny?
    A: The answer is 8


    Q: There were nine computers in the server room. 
    Five more computers were installed each day, from monday
    to thursday. How many computers are now in the server room?

Model completion:

A: The answer is 29

Start playing with the model: https://platform.openai.com/playground

LLMs are Stochastic Compilers

Yes, the LLMs can compile your instructions and solve the task. BUT…
- They are not deterministic.
- They fail

But they listen if you talk nicely to them:

Different prompting (programming) styles

So far, we have seen two different programming styles:
- Specification with instructions
- Specification with examples
We also saw that LLMs are stochastic, we may have to try several “variants” of the program to get the right one.

Regular programs also come in various flavors:

Stylistic differences

  # Good naming and formatting
  def calculate_area(width, height):
      return width * height

  # Poor naming and formatting
  def calc_a(w,h):return w*h

Implementation differences

  # Using a set to remove duplicates, more efficient and concise
  def remove_duplicates(numbers):
      return list(set(numbers))

  # Using a loop to remove duplicates, less efficient and more complex
  def remove_duplicates_using_loop(numbers):
      unique_numbers = []
      for number in numbers:
          if number not in unique_numbers:
              unique_numbers.append(number)
      return unique_numbers

Let’s go back to our example of math reasoning

Text prompt

Q: Jason had 20 lollipops. He gave Denny some lollipops. 
    Now Jason has 12 lollipops. How many lollipops did Jason
    give to Denny?
    A: The answer is 8


    Q: There were nine computers in the server room. 
    Five more computers were installed each day, from monday
    to thursday. How many computers are now in the server room?</code></pre>

Model completion:

A: The answer is 29

But we don't have to use plain boring text always!

We can supply examples of text → Python program.
LLM is prompted to generate code (Python). You can then run the Python script using a runtime!
- Opens up tons of possibilities.
- The Python program can call sympy, matplotlib, sklearn…

Code prompt!

# Q: Olivia has $23. She bought five bagels for $3 each. How much money does she have left?
# solution using Python:

def solution():
    """Olivia has $23. She bought five bagels for $3 each. How much money does she have left?"""
    money_initial = 23
    bagels = 5
    bagel_cost = 3
    money_spent = bagels * bagel_cost
    money_left = money_initial - money_spent
    result = money_left
    return result



# Q: There were nine computers in the server room. Five more computers were installed each day, from monday to thursday. How many computers are now in the server room?
# solution using Python:

def solution():
    """There were nine computers in the server room. Five more computers were installed each day, from monday to thursday. How many computers are now in the server room?"""
    computers_initial = 9
    computers_per_day = 5
    num_days = 4  # 4 days between monday and thursday
    computers_added = computers_per_day * num_days
    computers_total = computers_initial + computers_added
    result = computers_total
    return result

Summary and Key Takeaways

Language models like GPT-3.5/4/ChatGPT can be thought of as compilers that interpret prompts at various levels of abstraction.
Abstractions in programming languages evolve, with examples ranging from machine code to Python.
Prompts can be specified using different programming styles, such as instructions or examples.
Language models are stochastic compilers, requiring trial and error to produce the desired output.
Alternative forms of input, like code, can be used to achieve more precise results from the model.

Interactive Demos and Examples

Advanced Prompting Techniques

A nice survey of prompting. A great read especially if you are interested in understanding where NLP was before prompting.
Another great blog from Lilian on recent prompt engineering techniques.

Libraries

Our recent work on LLMs

Acknowledgements

The blog was originally written for a tutorial conducted at the CMU-LTI Seminar. Thanks to the organizers for the opportunity!
Thanks to Adithya Pratapa for proofreading the first draft.
Thanks to GPT-4 for generating some examples for this blog.

Notes on Weight Initialization for Deep Neural Networks

Mon, 01 Apr 2019 00:00:00 +0000

Tl; dr
Outline
Motivation
- Investigating the Sequence of Multiplications
- A first intuitive solution
Xavier Initialization
Why \(\sqrt(512)\)? | Intuition
Why \(\sqrt(512)\)? | Proofs

Tl; dr

Neural networks involve long sequence of multiplications, usually between a matrix and a vector, say \(a*x\). The result of this sequence of multiplications will either have a huge magnitude or be reduced to 0. We can divide \(a\) by a number (scaling_factor) to scale down its magnitude to the right level. Proper init strategies help us in finding a good scaling_factor.

Outline

The problem of weight initialization is motivated by a simulation. The real cause of ill-behaved multiplication output is identified, and scaling the weight matrix using the Xavier initialization is presented as a solution. The rest of the writeup then provides experiments and proofs to explain why such an initialization worked. The blog originated from a discussion during part 2 of the Fast AI course (Spring 2019 session), and parts of simulations are taken from the course notebooks.

Motivation

Training (and inference) of a neural network involves a bunch of operations, and one of the most common of these operations is multiplication. Typically, the multiplication happens between matrices. In the case of deep neural networks, we end up with a long sequence of such multiplications.

input = x
output = input
for layer in network_layers:
	output = activation(output * layer.weights + layer.bias)

Investigating the Sequence of Multiplications

To begin our investigation, let’s take a random input vector, x and a random matrix a. Note that the numbers are sampled from a normal distribution with a mean 0 and variance 1 or as it is popularly known, \(\mathcal{N}(0, 1)\).

We’ll multiply the vector x and the matrix a 100 times (as if the network had 100 layers), and see what gets out on the other side. Note that we don’t use any activation function for the sake of simplicity.

x = torch.randn(512)
a = torch.randn(512,512)
for i in range(100):
	x = a @ x
x.mean(),x.std()

output:

(tensor(nan), tensor(nan))

x has a huge magnitude! It seems like the multiplication snowballed, and the magnitude of x increased with each step, finally pushing the mean of a*x out of the limits of numbers in python (note that we are feeding x back to a @ x).

A first intuitive solution

Intuitively, since the product of a and x is becoming large, we may start by reducing the magnitude of the matrix a. The hope is that because of a smaller a, the product a * x won’t shoot up in magnitude. Thus, we divide our matrix (i.e., each element of the matrix a) by a scaling_factor of 100, and repeat the process.

scaling_factor = 100
x = torch.randn(512)
a = torch.randn(512,512) / scaling_factor
for i in range(100): 
	x = a @ x
x.mean(), x.std()

(tensor(0.), tensor(0.))

So we did solve the problem of magnitude explosion, only to create another: the output now vanishes to 0.

Xavier Initialization

We saw that using a scaling_factor of 100 didn’t quite work. It reduced the product to 0. We had started with the problem of the magnitude exploding to infinity, and the scaling brought it down to 0. Surely, the right solution lies somewhere in the middle. That’s exactly what the Xavier initialization does: it helps us in finding a scaling factor that would get it right.

The Xavier initialization suggests using a scaling factor of \(\sqrt(n\_in)\), where n_in is the number of inputs to the matrix (or the dimension that’s common with the vector the matrix is being multiplied with).

In our case, the number of inputs to the matrix \(a\) is 512. Thus, the scaling factor should be \(\sqrt(512)\). In other words, if we divide our matrix a by \(\sqrt(512)\), we should not see either vanishing or exploding magnitudes. Let’s see if the Xavier init helps:

import math
scaling_factor = math.sqrt(512)
x = torch.randn(512)
a = torch.randn(512,512) / scaling_factor
for i in range(100):
	x = a @ x
x.mean(), x.std()

(tensor(0.0429), tensor(0.9888))

The magnitude of the product hasn’t exploded or vanished. In fact, the output has a nice mean (close to 0) and standard deviation (close to 1). Recall that the input was actually sampled from such a distribution. In a way, our solution managed to retain the distribution of the inputs. That’s a really nice thing, because now we can perform a large number of such multiplications.

Putting things in context, this translates to being allowed to train really deep neural networks. Note that Xavier initialization is sufficient to solve the problem in this case because we did not use any activation function. If we had used, say a ReLu, the more recent Kaiming Initialization would have been more effective. So why did this work? What is so special about \(\sqrt(512)\) as a scaling factor?

Why \(\sqrt(512)\)? | Intuition

Before we start, let us look closely at our simulation, particularly the following line:

x = a @ x

Note that we are not changing a at all. Thus, the only element that can cause trouble is x, since it’s being updated. Seems like at some point in the multiplication sequence, x starts getting high values, and thus the subsequent multiplications keep making things worse. To closely examine this phenomenon, let us denote the product of a and x by y.

\[y = a * x\]

As in our running examples, if a is a matrix of size 512 x 512, and x is a vector of size 512, then output y is a vector of the size 512.

To be more explicit, one element of y is calculated as follows:

\[y_{i} = a_{i,0} x_{0} + a_{i,1} x_{1} + \cdots + a_{i,n-1} x_{n-1} = \sum_{k=0}^{n-1} a_{i,k} x_{k}\]

As we saw above, something goes wrong with the y values. That something is the following:

To compute one element of y, we add 512 products of one element of a by one element of x. What’s the mean and the variance of such a product? As we show later, as long as the elements in a and the elements in x are independent (which they are in this case; one doesn’t affect the other), the mean is 0 and the variance is 512. That is, each element of y is now taking the values as if it was picked \(\mathcal{N}(0, 512)\)! This can also be seen experimentally as in the following code snippet. To avoid one-off errors, we repeat the experiment for 10000 iterations.

mean, var = 0.0, 0.0
n_iter = 10000
n_dim = 512
ys = []
for i in range(n_iter):
	x = torch.randn(n_dim)
	a = torch.randn(n_dim) #just like one row of a
	y = a@x
	mean += y.item()
	ys.append(y.item())
mean/n_iter, torch.tensor(ys).var(), torch.tensor(ys).std()

(-0.13198307995796205, tensor(513.4638), tensor(22.6597))

In other words, each element of y is now picked from an erratic distribution, and that’s happening because we are adding a product of 512 elements, each picked from \(\mathcal{N}(0, 1)\). We keep feeding these y elements again in the loop as the input, and thus things go haywire soon.

Now, if we scale the weights of the matrix \(a\) and divide them by \(math.sqrt(512)\), we will be picking elements of \(a\) from a normal distribution with \(0\) mean and variance = \(512\) or \(\mathcal{N}(0, 512)\) (see the next section for a proof).

This scaling will, in turn, give us a distribution of \(y\) in which each element has 0 mean and std = 1, thus allowing us to repeat the product has many times as we want. This is NOT different from the intuitive solution we had discussed earlier. We were right in guessing that scaling one of the participants in the product may help, and it did. Xavier init helped us in finding out the exact magnitude of the scaling factor: \(\sqrt(512)\), instead of what we had initially used: 100.

mean, var = 0.0, 0.0
n_iter = 10000
n_dim = 512
ys = []
for i in range(n_iter):
	x = torch.randn(n_dim)
	a = torch.randn(n_dim) / math.sqrt(n_dim) #just like one row of a
	y = a@x
	mean += y.item()
	ys.append(y.item())
mean/n_iter, torch.tensor(ys).var(), torch.tensor(ys).std()

(-0.00671036799326539, tensor(1.0186), tensor(1.0092))

It works, and each element of y (and y as a whole) now has mean 0 and variance/std 1. We can thus keep multiplying the output y with a repeatedly, without worrying about things changing a lot.

Why \(\sqrt(512)\)? | Proofs

We are given that \(x\) and \(a\) are from a normal distribution with mean = 0 and variance = 1 (or \(\mathcal{N}(0, 1)\)). That is, to create \(x\), we pick 512 random numbers from \(\mathcal{N}(0, 1)\) (see). Similarly, to create \(a\), we pick (512 * 512) random numbers from \(\mathcal{N}(0, 1)\) . Then, the \(i^{th}\) element of \(y\) is calculated by multiplying 512 elements of \(a\) (i.e. \(a[i]\)) with 512 elements of \(x\).

\[y_{i} = a_{i,0} x_{0} + a_{i,1} x_{1} + + a_{i,511} x_{511} = \sum_{k=0}^{511} a_{i,k} x_{k}\]

1. Proof that \(Y \sim \mathcal{N}(0, 512)\)

Let \(A\), \(X\) and \(Y\) be the random variables from which \(a\), \(x\) and \(y\) are sampled respectively. We know that one element of \(Y\) is created by multiplying 512 elements from \(A\) and \(X\) with each other. That is, we sample 512 elements from \(A\), 512 elements from \(X\), multiply them element by element, and add them. Thus, so far we have:

\[\begin{aligned} & A \sim \mathcal{N}(0, 1) \\ & X \sim \mathcal{N}(0, 1) \\ & E[A] = 0 \\ & E[X] = 0 \\ & Var[A] = Std[A] = 1 \\ & Var[X] = Std[X] = 1 \\ \end{aligned}\]

and \(\begin{aligned} Y = \sum_{k=0}^{511} A*X \end{aligned}\)

Let’s start by calculating the mean of Y

1.1 Expectation (Mean) of Y

\[\begin{aligned} E[Y] & = E[AX] \\ & = E[A] * E[X] = 0 & (\text{A and X are independent, and E[A] = E[X] = 0}) \end{aligned}\]

(See properties of expectation)

1.2 Variance of Y

We know that \(Y\) is created by adding 512 elements sampled from \(A*X\). Thus, let’s first calculate the variance of \(A*X\). That is, what would be the variance if we pick one element randomly from \(A\) and \(X\) and then multiply them?

\[\begin{aligned} Var[AX] & = Var(A)*(E(X))^2 + Var(X)*(E(A))^2 + Var(A)*Var(X) & (\text{A and X are independent}) \\ & = Var(A) * Var(X)\\ & = 1 \end{aligned}\]

(Reference for the variance property)

We know that Y is formed by summing 512 such elements or

\[\begin{aligned} Y = \sum_{k=0}^{511} A*X \end{aligned}\]

Thus

\[\begin{aligned} Var[Y] & = Var[\sum_{k=0}^{511}A * X] \\ & = \sum_{k=0}^{511} Var[AX] &(\text{A and X are independent}) \\ & = \sum_{k=0}^{511} 1 &(\text{Var[AX] = 1 from above}) \\\\ & = 512 \end{aligned}\]

(Reference)

In other words, \(Y \sim \mathcal{N}(0, 512)\) which is terrible, since Y now varies a lot! The experiment is reproduced below for ready reference. Each of the ys have a large variance! As they are fed to the subsequent layers, the product can only get worse, as we’ve seen.

mean, var = 0.0, 0.0
n_iter = 10000
n_dim = 512
ys = []
for i in range(n_iter):
	x = torch.randn(n_dim)
	a = torch.randn(n_dim) #just like one row of a
	y = a@x
	mean += y.item()
	ys.append(y.item())
mean/n_iter, torch.tensor(ys).var()

(-0.10872242888212204, tensor(514.2963))

2. Proof that Y is \(\sim \mathcal{N}(0, 1)\) when A \(\sim \mathcal{N}(0, 1 / 512)\)

If we scale the weights of the matrix \(a\) and divide them by this \(\sqrt(512)\), we will be picking elements of \(a\) from a distribution with \(0\) mean and variance = (1 / 512) i.e. \(\mathcal{N}(0, 1 / 512)\). This will in turn give us a distribution of \(y\) in which each element has 0 mean and std = 1, thus allowing us to repeat the product as many times as we want (or in other words, make our network deeper).

We will now prove that dividing a by \(\sqrt(512)\) leads to \(Y\) getting a better distribution.

Now we have

\[\begin{aligned} & A \sim \mathcal{N}(0, 1 / 512) \\ & X \sim \mathcal{N}(0, 1) \\ & E[A] = 0 \\ & E[X] = 0 \\ & Var[A] = 1 / 512, Std[A] = 1 / \sqrt(512) \\ & Var[X] = Std[X] = 1 \\ \end{aligned}\]

2.1 Expectation (Mean) of Y

\[\begin{aligned} E[Y] & = E[AX] \\ & = E[A] * E[X] = 0 \ (\because E[A] = E[X] = 0) \end{aligned}\]

2.2 Variance of Y

As before, let’s first calculate the variance of \(AX\). That is, what would be the variance if we pick one element randomly from \(A\) and \(X\) and then multiply them?

\[\begin{aligned} Var[AX] & = Var(A)*(E(X))^2 + Var(X)*(E(A))^2 + Var(A)*Var(X) \\ & = Var(A) * Var(X) = 1 / 512 \end{aligned}\]

Now,

\[\begin{aligned} Y = \sum_{k=0}^{511} A*X \end{aligned}\]

Thus,

\[\begin{aligned} Var[Y] & = Var[\sum_{k=0}^{511}A * X] \\ & = \sum_{k=0}^{511} Var[AX] &(\text{A and X are independent}) \\ & = \sum_{k=0}^{511} 1 / 512 &(\text{Var[AX] = 1 from above}) \\\\ & = 1 \end{aligned}\]

In other words, \(Y \sim \mathcal{N}(0, 1)\) which is what we wanted! Let’s do an experiment to make sure this holds:

mean, var = 0.0, 0.0
n_iter = 10000
n_dim = 512
ys = []
for i in range(n_iter):
	x = torch.randn(n_dim)
	a = torch.randn(n_dim) / math.sqrt(n_dim) #just like one row of a
	y = a@x
	mean += y.item()
	ys.append(y.item())
mean/n_iter, torch.tensor(ys).var(), torch.tensor(ys).std()

(-0.008042885749042035, tensor(0.9856), tensor(0.9928))

Works! Each element of y will thus be sampled from a well behaved distribution. Here is the original simulation with the fix for a quick reference:

scaling_factor = math.sqrt(512)
x = torch.randn(512)
a = torch.randn(512,512) / scaling_factor
for i in range(100): 
	x = a @ x
x.mean(), x.std()

(tensor(0.0121), tensor(1.1693))

Summary

The operation of sequenced multiplication of matrices lies at the core of neural networks. We see that without proper initialization, inputs sampled from well-behaved distribution \(( \mathcal{N}(0, 1))\) will vanish (over-scaling) or explode (under-scaling). Dividing weight matrix by \(\sqrt(num\_inputs)\) (num_inputs = 512 in the running example), known as Xavier Initialization, helps in ensuring that the output of each of the multiplications is well-behaved, thus ensuring that the sequence of multiplications yields a reasonable output at each step. While Xavier initialization puts us on the right track, Kaiming Initialization provides the optimal scaling factor when ReLu is used as an activation (non-linearity) between multiplications in the network.

The Curious Case of the Missing US 101 Exits

Wed, 31 Jan 2018 00:00:00 +0000

TL; DR

Exits on the US 101 are not numbered sequentially, and are based on distance. Two subsequent exits that are 5 miles apart will have exit numbers that differ by 5. Various visualizations show that the exits are densely packed in the cities and become increasingly sparse as we move north.

TL; DR
Background
Collecting Exit Numbers
Visualizing the Gaps
So, What’s Happening?
- The Large Missing Patch
- The Missing Exit 413

Background

I’ve known about the existence of the US 101 for about 2 years now; hopping on every now and then for moving around the Bay area. I’ve never paid close attention to the exit numbers and always assumed that the exits are assigned numbers in the serial order. For example, exit 400 will be between exits 399 and 401 etc. I was wrong.

The truth dawned upon me as I started noticing the exit numbers while driving from Sunnyvale to Foster City a couple of nights ago. To kill the boredom and monotony, I started to try and calculate the square of an exit number before the next exit (the exits on this particular stretch are centered around 400, so it’s easier; 405² = (400 + 5)² etc.). It didn’t take long before it became clear to me that not all the exits were present. Particularly shocking was the fact that the anomaly (?) existed quite close to the home as well. I passed exit 412 (Ralston/Oracle) and proceeded to take exit 414 (Hillsdale) for Foster City; there was no Exit 413. I’ll be damned.

Collecting Exit Numbers

The list of exits was easy to obtain. I scraped these links and obtained a list of the northbound and southbound exits (I’ve posted a cleaned list of exits here). For this analysis, I used only the exit numbers, and removed the alphabetical suffixes (414A, 414B becomes a single entry, 414). The exits obtained were between 1 to 734, and included California only (the US 101 extends to Oregon and Washington as well). I verified the correctness of the lists by spot checking a few exits using Google maps. It’s worth mentioning that there are a few instances where an exit is only northbound or southbound. For example, exit 415 (Kehoe Avenue), between 92 and 3rd Avenue San Mateo only happens on northbound 101. However, 95% of the exits have both a north and a south instance, and thus I’ll focus only on the northbound exits in this post in the interest of brevity. Finally, note that the exit numbers start from the south and increase as we move north: from exit 1 (4th Street in LA) to exit 734 (Patricks Point Drive, Trinidad).

Visualizing the Gaps

The list of exits begins at 1 and ends at 734, and has 342 elements. Thus, there are 392 exits (53.40%) missing in the sequence. Here is a random snippet from the list of exits:

[381, 382, 383, 384, 385, 386, 388, 389, 391, 392, 393]

From the sample, we can see that the exits 387 and 390 don’t exist. I’ll apply some basic transformations on the list of exits to make it easier to see what’s going on. To illustrate the transformations, I’ll use the running example of a smaller list, [1, 2, 6] (exits 1, 2 and 6 are present, exits 3, 4 and 5 are absent). The transformations are discussed next.

Transformations

1. Present/Absent Array

Create an array by placing a “1” for every exit that’s present, and a “0” for every exit that’s absent.

E.g. [1, 2, 6] => [1, 1, 0, 0, 0, 1].

2. Running Sum

Create the present/absent array, but put a “1” for the missing exits and a “0” for the present ones.
Record a running sum of the elements in the new array, but reset the running sum every time a “0” is encountered.
The running sum thus captures the intensity of gaps. Longer gaps in the exits will lead to larger values of the running sum.

E.g. [1, 2, 6] => [0, 0, 1, 1, 1, 0] => [0, 0, 1, 2, 3, 0]

3. Binning

Create the present/absent array.
Decide on a bin size, say N.
Create another array from the present/absent array, by summing every N elements.

E.g. for N = 2, we get: [1, 2, 6] => [1, 1, 0, 0, 0, 1] => [2, 0, 1]

The visualizations are presented next. All the visualizations were created using standard matplotlib functions (particularly, imshow). The trendline was generated using numpy polyfit

Spectrogram with Present/Absent Array

The following plot was generated using the present/absent array. There is a black line for every exit that’s present (i.e., for every 1 in the present/absent array).

As can be clearly seen from the plot, the exits become sparser as we move north. There seems to be a large gap close to exit 550, with about 50 exits missing! Exits closer to the starting point in LA seems to be densely packed.

Spectrogram with Running Sum

The above plot uses the value of the running sum to decide on the color of the line that’s plotted. No gaps (running sum = 0) are black, and the colors change from red to blue, yellow and ultimately white as the value of the running sum increases. The plot reinforces the message about sparsity towards the north. A neat insight that this plot makes more obvious is the distinction between cities and wildernesses. The black patches are the densely populated localities, and the red, blue, yellow patches are the gaps. The large gap of around 50 missing exits near exit 550 comes out very distinctly. We also see that exits closer to the starting point (LA) and around exit number 400 (Bay Area!) are pretty densely allocated as opposed to rest of the freeway.

Trendline

The next plot shows the binned present/absent array with a bin size of 25, along with a trend. The plot reinforces our notes from the above sections that the number of exits becomes smaller and smaller as we move north.

So, What’s Happening?

It turns out that these exits are not sequential, and are based to serve as a basis for measuring the distance. From the relevant wiki

Freeway exits in the United States are usually numbered in two formats: distance-based and sequential.

So, that’s the mystery. The exit numbers are derived based on the distance between the consecutive exits. Finally, let’s see if the findings are backed by maps.

The Large Missing Patch

All of the plots revealed a large gap near the northern end of the 101. I looked up the exact exits from the list of northbound exits, and it turns out that exit 557 is followed by exit 609. That’s 52 exits missing, which should mean a distance of around 52 miles. That seems to be the case, as indicated by this snippet taken from this route.

The Missing Exit 413

Similarly, it seems that the distance between the Hillsdale and the Oracle exit is about 1.6 miles, and the distance between Oracle and the San Mateo exit is 2.3 miles, which explains the missing exit 413!

Thanks for making it this far, and please don’t hesitate to share your questions/comments. Thanks to Saugata Chowdhury for the helpful brainstorming session.

Training Char-RNNs for Transferring Name Styles

Sat, 18 Nov 2017 00:00:00 +0000

Introduction
Looking at the Data
- Top 5 Most Popular Names
- Name Length Distributions
Input Representation
Model
Transferring Name Styles
Results

Introduction

Dataset

A few days ago I was enticed into thinking if there is an app for generating names (~~no, not a startup idea~~). You know, something along the lines of “generate 10 Indian names”. I didn’t know the answer, but it got me thinking about names. Before the flame of this random musing could die down, I found a dataset of names, thanks to mbejda. The dataset consists of Hispanic, Indian, Caucasian and African American names. I had the dataset, so something had to be done.

RNNs

Now perhaps you have read Karpathy’s fantastic blog on RNNs and what they do.

Here is one of quotes from the blog:

We’ll train RNNs to generate text character by character and ponder the question “how is that even possible?”

Yes, it’s pretty mind-blowing. The blog goes into the details of how it’s done. Here’s a short summary of what RNNs do: RNNs can take a bunch of text, say T, and learn to generate new text that would seem to be taken from T. For example, we can train an RNN on all the works of William Shakespeare, and the RNN can, in turn, generate new text that would seem to be written by Shakespeare. The blog I’ve linked to contains this and many other interesting examples. The way RNNs do this is by learning to “predict the next character” in the sequence given the hitherto seen characters.

Putting it together, I had:

A model that can learn to generate new samples from a piece of text.
A corpus of names.

So it would not be very inspiring if I say that we can use RNNs to generate new names. Yes, we can, and yes, I did. I got an “Indian Name Generator” generating Deepaks and Nehas, a “Caucasian Name Generator” generating Michaels and Jennifers and so on. It was something, but as I said, not very interesting.

Cross Seeding

Now, I didn’t feel like throwing all these RNNs away, so wondered what would happen if I feed, say, the “Indian Name Generator” with a first few characters from a Caucasian name, and let it generate the rest? Will it try to create a name that sounds Indian? So I ran a bunch of these experiments, and present the somewhat more interesting results in this post. I also used seeds from unconventional names, like those of Pokémons and wrestlers. It was fun to see all these different RNNs take a stab at creating a name that sounds to be from their domain:

name	seed	african_american	caucasian	hispanic	indian	all_races
undertaker	underta	undertall nix#	undertan starlir#	underta romero#	undertala#	undertayshawn king#
aman madaan	aman mad	aman madadenis#	aman madich#	aman madro l gonzalez#	aman madhkaran#	aman madha#
jose luis	jose l	jose l graham#	jose l ramirez#	jose l morales#	jose lal sharma#	jose l rodriguez#
hideyoshi	hideyo	hideyon u bennett#	hideyo g morio#	hideyordo rodriguez#	hideyohar sharma#	hideyon d brown#
dan fineman	dan f	dan f briggs#	dan f witharr#	dan flekrez#	dan farjat saini#	dan francersiii#
hulk hogan	hulk ho	hulk hornes#	hulk howstie#	hulk hoelles.maldonado#	hulk holoo chand singh#	hulk holu#

This post has 6 sections: The introduction just got over. We’ll now take a quick look at the data, followed by a discussion on how the input is converted to a representation that can be used for training these name generators. We’ll then look at some details of the model, how the predictions are done and the name styles transferred, and finally present the results.

The code, with cleaned + processed dataset, and notes on how to run the training and scoring processes, is located here.

Looking at the Data

Although there is no limit to the number of different analysis we can run, we’ll present only two here in the interest of space (and the attention span): i) the most popular names and ii) the name length distributions. Looking at the most popular names will give us a feel for the dataset, and the name length distributions were added to add more plots and make this post look fancier (and it’s used somewhere down the line, too).

Top 5 Most Popular Names

The following tables list the top 5 most popular names for each of the races. Please note that the first and the last names are listed separately (e.g., Latoya Williams is not the most popular African-American female name; Latoya is the most popular first name for African-American females, and Williams is the most popular last name for African-American females).

African American
Female First Names	Female Last names	Male First Names	Male Last Names
latoya	williams	michael	johnson
ashley	johnson	james	brown
patricia	brown	anthony	jones
angela	smith	willie	jackson
mary	jackson	robert	davis

Caucasian
Female First Names	Female Last Names	Male First Names	Male Last Names
jennifer	smith	michael	johnson
amanda	brown	james	rodriguez
kimberly	williams	robert	davis
jessica	miller	david	jones
ashley	johnson	john	brown

Hispanic
Female First Names	Female Last Names	Male First Names	Male Last Names
maria	rodriguez	jose	rodriguez
melissa	gonzalez	juan	garcia
jennifer	rivera	luis	martinez
gloria	perez	carlos	rivera
elizabeth	garcia	jorge	hernandez

Indian
Female First Names	Female Last Names	Male First Names	Male Last Names
smt	devi	deepak	kumar
pooja	pooja	rahul	singh
smt.	kumari	amit	sharma
jyoti	jyoti	ram	lal
kumari	bai	sanjay	ram

Name Length Distributions

The name length distributions are next plotted for each of the races. It seems like short names (perhaps without a surname) are popular among Indians, giving rise to the minor mode in the distribution. Hispanic names tend to be longer, as indicated by the fat tail following the mean. 15-ish seems to be the most popular name length across the races (here’s one thing you can take away from the post).

Input Representation

In this section, we will spend some time looking at how do we take a bunch of these names and convert them into a form that can be fed to an RNN. We will get to that representation in three steps: encoding, standardization, and embeddings.

a) Encoding

We convert each character in a (string) name to a number using the following mapping:

Character	Encoding
a-z	0-25
” “ (Space)	26
”#” (End of Name)	27
. (Invalid Character)	28

All the names are converted to lowercase English alphabets, with a space separating different components of a name. Every name ends with a special name end character (“#”). Every other character is mapped to a “.”, an invalid character substitute. For example, “joe” would be converted to [9 14 4 27].

b) Standardization

Name lengths are anything but a constant, as seen from the length distributions. However, we are using “unrolled” RNNs, and thus we need to fix on a maximum name length. Names longer than the maximum name length will be truncated, and names shorter than the maximum name length will be padded with “.” (invalid character). As discussed, we assume that every name ends with a “#,” the name end character. All of this happens in the following piece of code:

def encode_and_standardize(name):
   name = name + CharCodec.NAME_END #add the end of the name symbol for everyname
   name = CharCodec.encode(name) #encode
   if name_len >= CharCodec.max_name_length:
       truncated_name = name[:(CharCodec.max_name_length - 1)]
       truncated_name.append(CharCodec.char_to_class[CharCodec.NAME_END]) #must attach the name end
       return np.array(truncated_name, dtype=np.int32)
   else:
       padded_name = np.empty(CharCodec.max_name_length, dtype=np.int32)
       padded_name.fill(CharCodec.INVALID_CHAR_CLASS)
       padded_name[:name_len] = name
       return padded_name

Note that we retain the name end character (#) even after the truncation.

So how do we arrive at the max_name_length? A simple way to do that is fix on a large number, like 100. However, that would mean that our network will be wider than we perhaps want (most of the names will be smaller than 100 characters). This will lead to lots of wasted computation and slower training times (give it a try!). Or, we could plot a distribution of the name lengths and pick something simple. We’ve already done that, and as you can see, it seems like 25 will cover most of the cases, and we add one space to accommodate the name end marker, “#.” Thus, all the names are standardized to length 26. At this point, the string name “joe” has become an array of 26 numbers, [ 9 14 4 27 28 28 28 28 28 28 28 28 28 28 28 28 28 28 28 28 28 28 28 28 28 28].

c) Embeddings

So far, we have standardized each name to a fixed length, added a character to mark the end of the name, and encoded the name from an array of chars to an array of integers.

If you don’t know what embeddings are at all, I recommend checking out this or this link.

The tl;dr of the technique is that each of the characters is mapped to an array of numbers. The array is called an embedding, and the length of the array is the dimensionality of the embedding. For example, if we choose to use 5-dimensional embeddings, it just means that every character is mapped to a 5-dimensional real vector. We can start with pre-trained embeddings, or learn them as part of the training process, which is what our model does. In our setting, that means that we will hopefully learn embeddings that makes it easier for us to predict the next character in the name. With 5-dimensional embeddings, “joe” will now be a matrix of 26 rows and five columns.

The following three lines of code is all that’s needed add embeddings. embeddings is a matrix, which has one row for each character in our vocabulary. Concretely, if we had used 5-dimensional embeddings, the embeddings matrix would have the dimensions 29 x 5; each of the characters would have a corresponding array of length 5. tf.nn.embedding_lookup takes in the input, which is batch_size x max_name_length, and maps (looks up) each character in the standardized name to an embedding, to yield an input with dimensions batch_size x max_name_length x n_embeddings.

names = tf.placeholder(tf.int32, shape=(None, max_name_length), name="input")
embeddings = tf.Variable(tf.random_uniform([vocab_size, n_embeddings], -1.0, 1.0))
embedded_names = tf.nn.embedding_lookup(embeddings, names) #(?, max_name_length, n_embeddings)

Tensorboard can help in visualizing embeddings using PCA and t-SNE. The t-SNE visualization of the embedding matrix from the Indian name generator is as follows. As you can see, the vowels are all close to each other, which hints at the fact that the learned embeddings wrap some linguistic properties of the names pertaining to the race, and are perhaps useful in generating new ones.


Embeddings learned by the Indian Name Generator visualized using Tensorboard

Model

The particular form of RNNs that we use for this exercise is an LSTM. This book has a pretty good explanation of the LSTMs, and this neat blog post is another standard reference for the topic. A high-level overview of the model follows. Each character in the normalized name is converted to the corresponding embedding vector, and the entire input name becomes a matrix or name embeddings. The name embeddings are then fed to the first LSTM in the stack. The output from this first LSTM is fed to a second LSTM. The second LSTM is connected to a dense layer, which emits a logits vector of length 29. An argmax over the logits vector is used to calculate the loss and generation. A diagram of the computation graph generated by Tensorboard is as follows:


The overall model setup generated by tensorboard. The input Embeddings are fed to stacked LSTMs, which in turn feed to a dense layer

An instance of the model with made up numbers is shown below. It’s a replica of this diagram from this blog I’ve already linked to.


A sample instance of the model. The input name is “Amy#” (# being the end of the name character). At each step, the network is expected to predict the next character. Thus, at step 1, the expected output is “M”, the 2nd character. At step 2, the expected output becomes the 3rd character, Y. As explained in the encoding section, the characters after the “#”, “.”, are invalid characters added for padding.

A sample instance of the model. The input name is “Amy#” (# being the end of the name character). At each step, the network is expected to predict the next character. Thus, at step 1, the expected output is “M”, the 2nd character. At step 2, the expected output becomes the 3rd character, Y. As explained in the encoding section, the characters after the “#”, “.”, are invalid characters added for padding.

Transferring Name Styles

Since we have discussed a lot, let’s quickly recap before moving ahead. Some of the following may seem to be a repeat from the introduction because it sort of is.

Dataset: We have a dataset of names from different races.
Generator: We have discussed a model that can be trained to predict the next character given a sequence of characters. Let’s call this model the generator.

We train one generator per dataset. Thus, we have a model that has learned to predict the next few characters in an Indian name given the first few, and so on. We then seed each of these generators with a few characters from a name, say “Der” from “Derek,” and compare the results.

The prediction process is illustrated in the figure below, followed by the code.

Code:

res = seed
initial_seed_offset = len(seed)
for i in range(CharCodec.max_name_length - len(seed)):
    feats = CharCodec.encode_and_standardize(res).reshape(1, CharCodec.max_name_length)
    prediction = sess.run(model.prediction, feed_dict={names: feats})
    res += " ".join(CharCodec.decode(prediction[0])[i + initial_seed_offset - 1])
    if res[-1] == CharCodec.NAME_END:
        break

Results

The results are compiled in the following table. The first column is the name, the second the seed used as an initial input to the model. The subsequent columns list the names generated by the different generators using the given seed.

name	seed	african_american	caucasian	hispanic	indian	all_races
zhang wei	zhan	zhankhea l stencor#	zhane a nelson#	zhanole s estrada#	zhanna sankar#	zhanson e martin#

In the above example, the seed used is “zhan,” from the Chinese name “zhang wei.” All the networks take this tricky seed and generate a name that finally looks like a name from the given model. For example, the African-American generator yields “zhankhea l stencor” (the “#” is the end of the name marker), the caucasian generator yields “zhane a nelson” and so on.

Results from this test file are as follows. I hope the post was useful, please feel free to contact me for any questions/comments at amn.madaan@gmail.com.

Thanks!

name	seed	african_american	caucasian	hispanic	indian	all_races
derek vroom	de	derrick l curtis#	dennis r coleman#	dennis r suro#	deepak . pardeep#	derrick l brown#
derek vroom	derek	derek a clark#	derek a hardellin#	derek m sanchez#	derekha bhai#	derek a carter#
derek vroom	derek vr	derek vright#	derek vraworit#	derek vrkilla.carcial#	derek vrasha#	derek vrow#
dan fineman	da	david l morgan#	david l mccoy#	daniel carachure#	dalip shah#	david l brown#
dan fineman	dan f	dan f briggs#	dan f witharr#	dan flekrez#	dan farjat saini#	dan francersiii#
dan fineman	dan fine	dan finette tyrel#	dan fines#	dan finera.garcia#	dan fine#	dan fine#
carolyn kennedy	car	carlos e jr robinson#	carlos r reyes#	carlos a guerra#	carta#	carlos a cardona#
carolyn kennedy	carolyn	carolyn r miller#	carolyn a roe#	carolyno jr rodriguez#	carolynir#	carolyn r martinez#
carolyn kennedy	carolyn ken	carolyn kendry#	carolyn kendrick#	carolyn kendeles.rodriguez	carolyn kent#	carolyn kent#
michael zuckerberg	mich	michael a chapman#	michael a mckinney#	michael a martinez#	michael#	michael a mccormick#
michael zuckerberg	michael z	michael z chatman#	michael z baker#	michael z martinez#	michael zah#	michael z russell#
michael zuckerberg	michael zucke	michael zuckenne#	michael zuckellan#	michael zucke#	michael zuckena#	michael zuckelli#
gary yao	ga	gary l hayes#	gary w jr davis#	gabriel a romero#	gaurav sharam#	gary l baker#
gary yao	gary	gary l hayes#	gary w jr davis#	gary avila#	gary#	gary l baker#
gary yao	gary y	gary y jr brown#	gary yancey#	gary y avares#	gary yadav#	gary yadav#
pranay chopra	pra	pradel lucas#	praie r saliska#	pramie rivera#	pramod kumar tiwari#	pramod kumar tiwari#
pranay chopra	pranay	pranay s mccarter#	pranayoos n santana#	pranay a gonzalez#	pranayavara bijgut#	pranayan devi#
pranay chopra	pranay ch	pranay charlon#	pranay chess#	pranay chaveda#	pranay chawla#	pranay chand#
mayank bhardwaj	may	maydreona l miles#	maykel c calderon#	maynor padilla.estrada#	maya kumari#	maya devi#
mayank bhardwaj	mayank	mayank redison#	mayank j romero#	mayank h rodriguez#	mayank .aakash#	mayank pandey#
mayank bhardwaj	mayank bhar	mayank bharder#	mayank bharresse#	mayank bharque#	mayank bhardwaj#	mayank bhardwaj#
aman madaan	am	amos e robinson#	amanda l hatcher#	amalio rivera#	amit kumar singh#	amanda l hutchinson#
aman madaan	aman	aman j joseph#	aman p jr marshall#	aman f rivera#	aman saroj#	aman sharma#
aman madaan	aman mad	aman madadenis#	aman madich#	aman madro l gonzalez#	aman madhkaran#	aman madha#
priyanka lingawal	priy	priy loveranc#	priye j mergtone#	priy g martinez#	priyanka . pinki#	priyanka . pinki#
priyanka lingawal	priyanka	priyankauguson#	priyanka l monagan#	priyankaune#	priyanka . pinki#	priyanka . pinki#
priyanka lingawal	priyanka lin	priyanka linkley#	priyanka linto#	priyanka linorriz#	priyanka linditay#	priyanka linda#
uma sawant	um	umar a hassan#	umid r mann#	umeliar m sanchez#	uma . barkha#	uma devi#
uma sawant	uma s	uma stimer l jenkins#	uma s vancey#	uma s muniz#	uma shankar soni#	uma shankar#
uma sawant	uma saw	uma sawrm crow#	uma sawyornig#	uma sawillas#	uma sawan#	uma saweer#
ankur dewan	an	anthony l mcclain#	anthony r horn#	antonio r hernandez#	anil kumar sain#	anthony j perry#
ankur dewan	ankur	ankurick reed#	ankur m davis#	ankur mondales#	ankur kumar#	ankur sharma#
ankur dewan	ankur de	ankur dewin#	ankur dehalf#	ankur delapamill#	ankur devi#	ankur devi#
rahul gupta	ra	raymond l hopps#	randall l rice#	ramon a feliciano#	ram sagar#	raymond l holland#
rahul gupta	rahul	rahuleshor c robbin#	rahul a jr gonzalez#	rahul g martinez#	rahul sharma#	rahul sharma#
rahul gupta	rahul gu	rahul gudert#	rahul guenoa#	rahul gutineza#	rahul gupta#	rahul gupta#
zhang wei	zh	zhivago rodgers#	zhenya v deckarri#	zhecory colinza#	zhini devi#	zhenya santana#
zhang wei	zhan	zhankhea l stencor#	zhane a nelson#	zhanole s estrada#	zhanna sankar#	zhanson e martin#
zhang wei	zhang	zhang m forbeson#	zhang j henderson#	zhang g diaz.hernandez#	zhang depo#	zhang r johnson#
wang xiu ying	wan	wanda j malone#	wanda j dentoscionle#	wanda rodriguez#	wani bhavi#	wanda j davis#
wang xiu ying	wang x	wang x jamison#	wang x bailey#	wang x rodriguez#	wang xai d.o lageh ri par#	wang xiluseiia mond#
wang xiu ying	wang xiu	wang xiu douglas#	wang xiu r mccall#	wang xiu castillo#	wang xiu shaker#	wang xiu santing#
hideyoshi	hi	hillard j london#	hilario m gomez#	hiram a maldonado#	himanshu . annu#	himanshu . annu#
hideyoshi	hide	hideajrake redmon#	hidel f martinez#	hidel garcia#	hidee chaudhry#	hidena#
hideyoshi	hideyo	hideyon u bennett#	hideyo g morio#	hideyordo rodriguez#	hideyohar sharma#	hideyon d brown#
zhang xiu ying	zha	zharran e austin#	zhays j notz#	zhanole s estrada#	zhanna sankar#	zhayni essidion#
zhang xiu ying	zhang x	zhang x hall#	zhang x herty#	zhang x corrova#	zhang xeta#	zhang x i barraman#
zhang xiu ying	zhang xiu	zhang xiu donner#	zhang xiu r brack#	zhang xiu cardenas#	zhang xiu w.o#	zhang xiu partadill#
hashimoto	ha	harry l jr caine#	harold d minor#	harold a cardenas.valenci#	harish chand dua#	harold l harris#
hashimoto	hash	hashantae walleggter#	hashia l sannero.ferzill.c	hashus rodriguez.gutierre#	hashim#	hashim#
hashimoto	hashim	hashimmela j slove#	hashim naviguenteras#	hashimaer m gonzalez.matha	hashim#	hashim#
shinzo abe	sh	shantae d dennis#	shannon l bennett#	shawn d rodriguez#	shankar lal gopalani#	shane a brown#
shinzo abe	shinz	shinzella b dowers#	shinzeyne a scharden#	shinz a medina#	shinz yadav#	shinza delvacy#
shinzo abe	shinzo	shinzo iii hardy#	shinzo a dejesunnell.igos#	shinzo d rodriguez#	shinzo . raju#	shinzo . ammo#
maria guadalupe	mar	marcus d cook#	mark a beasley#	mario a martinez#	mariyam . sonam#	mark a barnes#
maria guadalupe	maria g	maria green#	maria g pierce#	maria g serrano#	maria garg#	maria g castillo#
maria guadalupe	maria guada	maria guada#	maria guadaguez#	maria guadal.pont#	maria guadal#	maria guada#
jose luis	jo	john l graham#	joseph a bernston#	jose a correa#	joyti sharma#	joseph a brown#
jose luis	jose	joseph l henry#	joseph a bernston#	jose a correa#	josender sharma#	joseph a brown#
jose luis	jose l	jose l graham#	jose l ramirez#	jose l morales#	jose lal sharma#	jose l rodriguez#
veronica	ve	vernon l hankerson#	vernon r locke#	veronica hernandez#	veer bhan#	vernon l brown#
veronica	vero	veronica m butler#	veronica b jamison#	veronica hernandez#	veronika bhinder#	veronica l brown#
veronica	veroni	veronica m butler#	veronica b jamison#	veronica hernandez#	veronika bhinder#	veronica l brown#
juan carlos	ju	julius mcdaniels#	justin a moore#	juan c montalba#	julfi . sonu#	justin l barnette#
juan carlos	juan	juan p laster#	juan c martinez#	juan c montalba#	juan morji#	juan c castillo#
juan carlos	juan car	juan cardonas#	juan carranza#	juan carrillo#	juan carodhari#	juan cardona#
rosa maria	ro	robert l moore#	robert l bresteane#	roberto c munoz rivera#	rohit agrawal#	robert l jr marshall#
rosa maria	rosa	rosa a hobbs#	rosa m demerd#	rosa m torres#	rosa kalah#	rosa a johnson#
rosa maria	rosa ma	rosa mannt#	rosa martinez#	rosa mata#	rosa mani#	rosa massey#
francisco javier	fran	frank j brown#	frank e carballo#	francisco j mendez#	frana#	frank l jr bennett#
francisco javier	francisc	francisco a nunez#	francisco gonzales#	francisco j mendez#	francischan#	francisco j mendez#
francisco javier	francisco ja	francisco javen#	francisco jaza.torrez#	francisco jaigue#	francisco jain#	francisco jaramaz.ramos#
maria elena	ma	marcus d cook#	mark a beasley#	mario a martinez#	manish kumar mandal#	mark a barnes#
maria elena	maria	maria murray#	maria l west#	maria g serrano#	mariam mumtaz#	maria l lassinorino#
maria elena	maria el	maria ellis#	maria elweow#	maria elaramdo#	maria elam#	maria eller#
brianna	b	brandon m fleming#	brandon l holland#	bryan santana#	bharat bhushan#	brian k bradley#
brianna	bri	brian k brickhound#	brian k saitham#	brian n acosta#	brij mohan thakur#	brian k bradley#
brianna	brian	brian k brickhound#	brian k saitham#	brian n acosta#	brianti#	brian k bradley#
chloe	c	charles e hall#	christopher m brannon#	carlos a guerra#	chander shekhar#	christopher m mcclendon#
chloe	ch	charles e hall#	christopher m brannon#	christian rivera#	chander shekhar#	christopher m mcclendon#
chloe	chl	chlente chasbin#	chloe p johnson#	chlistonae a funsez#	chlepal#	chletu s jr marshall#
destiny	d	david l morgan#	david l mccoy#	daniel carachure#	deepak . pardeep#	david l brown#
destiny	des	desmond t baker#	desiree l barnett#	dessie santiago#	desh raj#	desmond t harris#
destiny	desti	destiny j rios#	destiny straton#	destin santiago#	desti#	destiny j brown#
jeremiah	je	jermaine l martin#	jeremy m morrison#	jesus m salinas#	jeet pal .jitu#	jeremy l coleman#
jeremiah	jere	jeremy a lee#	jeremy m morrison#	jeremy o burgos#	jerender singh#	jeremy l coleman#
jeremiah	jeremi	jeremiah d preister#	jeremiah m conner#	jeremias v gomez#	jeremi rana#	jeremiah d brown#
josiah	j	james l caldwell#	joseph a bernston#	jose a correa#	jai kishan gupta#	joseph a brown#
josiah	jos	joseph l henry#	joseph a bernston#	jose a correa#	josin jomon#	joseph a brown#
josiah	josi	josiah m beachem#	josian montanez#	josie a rivera#	josin jomon#	josiah m martinez#
undertaker	un	undray anderson#	uney s mendes#	unesto brito#	unknown . monu#	undrae m mcclellan#
undertaker	under	underially r ivy#	underico cabez.garcia#	under p maldonado#	under jain#	underick j collins#
undertaker	underta	undertall nix#	undertan starlir#	underta romero#	undertala#	undertayshawn king#
yokozuna	yo	yolanda y kryger#	yosmanis a cruz#	yovanny z bautista#	yogesh chahar#	yogesh chandra jo#
yokozuna	yoko	yokondrae o hurdolk#	yoko d chambers#	yokonnihua f seplano#	yoko#	yokoshania c carter#
yokozuna	yokozu	yokozull bowell#	yokozuma c adams#	yokozua l uerda#	yokozuddina#	yokozua j harris#
andre the giant	and	andre l arnold#	andrew j denny#	andres r guajardo#	andhav#	andrew j brown#
andre the giant	andre t	andre t mccray#	andre t mccoy#	andre t salazar#	andre topta#	andre t morgan#
andre the giant	andre the g	andre the gilmer#	andre the gorsom#	andre the garcia#	andre the gadar#	andre the getti#
big show	bi	billy r jr collier#	billy j ball#	billy e caballero reyes#	birender kuma .r yadav#	billy j banks#
big show	big	big a mitchell#	big p o carson#	big mariano#	big singh#	big nemi#
big show	big sh	big shardley j jr atturwow	big shulte#	big shergorio luz#	big shan#	big sharma#
hulk hogan	hu	hubert l hunt#	hugh walthall#	humberto m malagon#	husainpreet kour#	hugh a holt#
hulk hogan	hulk	hulk iv herrit#	hulk leath#	hulk rodriguez.gonzale#	hulk mohd#	hulk kumari#
hulk hogan	hulk ho	hulk hornes#	hulk howstie#	hulk hoelles.maldonado#	hulk holoo chand singh#	hulk holu#
picachu	p	patrick l lee#	paul a branch#	pedro j salazarlopez#	parveen kaur . simi#	paul j boyd#
picachu	pic	picky edmores#	pick d sansom#	picer g melendez#	picay . sonu#	pick d moran#
picachu	picac	picace lewis#	picacco turdlere u alvie#	picacio burgos#	picachiram akwal#	picace a james#
mewtwo	m	marcus d cook#	michael a mckinney#	mario a martinez#	manish kumar mandal#	mark a barnes#
mewtwo	mew	mewille m rodriguez#	mewald j donnynons#	mewel achevedo#	mewa aggarwal#	mewasui . sanjali#
mewtwo	mewt	mewtis f jr baken#	mewthed shewnert#	mewtel diaz#	mewta#	mewtilles dison#
jigglypuff	ji	jimmy l jr floyd#	jimmy d booher#	jimmy r torres#	jitender sehrawat#	jimmy l barnett#
jigglypuff	jiggl	jiggl jordan#	jigglan a layhorn#	jigglan r alvaro#	jiggla#	jiggl r porter#
jigglypuff	jigglyp	jigglyphal#	jigglyp t cedeno#	jigglype vernarda#	jigglypri . papa#	jigglypu g martin#
bulbasaur	bu	burnell mckenney#	buddy l cook#	bulfrano garcia#	budh prakash#	buddy l mccray#
bulbasaur	bulb	bulber l hargus#	bulbernon k manond#	bulbim m rolon#	bulbul#	bulbul#
bulbasaur	bulbas	bulbas jr littles#	bulbas.san s layvoro#	bulbashumald gasialisanosh	bulbas#	bulbas b rivera#
charizard	ch	charles e hall#	christopher m brannon#	christian rivera#	chander shekhar#	christopher m mcclendon#
charizard	char	charles e hall#	charles e jr hartsfield#	charles r murillo#	charan singh . minchu#	charles a brown#
charizard	chariz	charizelo m laster#	charizy a berry#	chariz l martinez#	chariz#	chariz c hill#

Everyone Can Write Bad Code

Thu, 15 Jun 2017 00:00:00 +0000

It’s Simpler than You Think

Writing bad code is simpler than you think. Yada-yadas will make you believe that it takes special skills to craft junk code, that only a “chosen few” among us are capable of doing it. They may trick you into thinking that it takes a lot of apathy, the ability to hand wave, the special gut feeling to make bad assumptions, and a keen, sharp focus on the short term (or smaller picture). Sounds like a lot, doesn’t it?

Well, to tell you the truth, it’s not that hard. For those of you who strive to write poor code, with a special focus on screwing up on big data jobs, I have compiled a set of action items that will help you in embarking on this journey.

A word of warning is in order. As you read through the 16 points, you may get demotivated. You may think you’ll never be one of them. To fight such dark thoughts, you may want to think about the developers you had written off as good; haven’t you seen them doing one of these? If they can do it, so can you!

Aman Madaan

The case for inference-time compute

Next token prediction is awesome

Next token prediction is not enough

Distribution Transformation Perspective

LLMs Are Stochastic Compilers

Or How to Think About Prompting with an Imprecise but Hopefully Helpful Analogy

Contents

The Evolution of Abstractions in Programming

Common thread

You are not teaching the task to the processor–you are just specifying the task differently. The processor already knows how to add two numbers.

Language Models Are Compilers

Fleshing out the analogy

The Expressiveness of Prompting

Specification with instructions

Generate a poem that rhymes with 'cat' and has 4 lines."

"Write a program in Python to test if `n` is prime"

Specification with examples

Prompt:

Feed the prompt to the model:

Example: math reasoning

Prompt

Model completion:

LLMs are Stochastic Compilers

Different prompting (programming) styles

Let’s go back to our example of math reasoning

Text prompt

Model completion:

But we don't have to use plain boring text always!

Code prompt!

Summary and Key Takeaways

Interactive Demos and Examples

Advanced Prompting Techniques

Libraries

Our recent work on LLMs

Acknowledgements

Notes on Weight Initialization for Deep Neural Networks

Tl; dr

Outline

Motivation

Investigating the Sequence of Multiplications

A first intuitive solution

Xavier Initialization

Why \(\sqrt(512)\)? | Intuition

Why \(\sqrt(512)\)? | Proofs

1. Proof that \(Y \sim \mathcal{N}(0, 512)\)

1.1 Expectation (Mean) of Y

1.2 Variance of Y

2. Proof that Y is \(\sim \mathcal{N}(0, 1)\) when A \(\sim \mathcal{N}(0, 1 / 512)\)

2.1 Expectation (Mean) of Y

2.2 Variance of Y

Summary

The Curious Case of the Missing US 101 Exits

TL; DR

Contents

Background

Collecting Exit Numbers

Visualizing the Gaps

Transformations

1. Present/Absent Array

2. Running Sum

3. Binning

Spectrogram with Present/Absent Array

Spectrogram with Running Sum

Trendline

So, What’s Happening?

The Large Missing Patch

The Missing Exit 413

Training Char-RNNs for Transferring Name Styles

Contents

Introduction

Dataset

RNNs

Cross Seeding

Looking at the Data

Top 5 Most Popular Names

Name Length Distributions

Input Representation

a) Encoding

b) Standardization

B. If the following `hugeData.flatMap().reduceByKey().groupByKey().map(_.toString)` throws a OOM, Spark must be a fad.