Skip to content

Navigation Menu

Sign in
Sign up

Add gpt2 implementations in python and c++ - #1

Open
ChinmayK0607 wants to merge 5 commits into
IvLabs:master from
ChinmayK0607:master
Open

Add gpt2 implementations in python and c++ #1
ChinmayK0607 wants to merge 5 commits into
IvLabs:master from
ChinmayK0607:master

Conversation

@ChinmayK0607

@ChinmayK0607 ChinmayK0607 commented Jul 8, 2025

Copy link
Copy Markdown
Member

Adds two files:

  1. train.cpp -> gpt2 implementation in c++
  2. train.py -> gpt2 implementation in python

Adds dataloaders as well as requirements file as well.

kamatajinkya2 commented Jul 8, 2025 •
edited
Loading

Copy link
Copy Markdown

General

  1. Prefer pyproject.toml over requirements.txt. Refer to Why Should I Choose pyproject.toml over requirements.txt for managing dependencies?
  2. Use a nested folder structure. Aka train.py, dataloader.py go inside src or llm101 or scripts folder. This will help in adding a test folder
  3. Implement unit tests (Not applicable in this instance, but just typing out)
  4. Have appropriate white spaces after a class of a function ends. You can use autoformatters like Black to achieve this.

To be continued...

Comment thread GPT2/dataloader.py
import numpy as np

# download the tiny shakespeare dataset
input_file_path = os.path.join(os.path.dirname(__file__), 'input.txt')

@kamatajinkya2 kamatajinkya2 Jul 8, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use if name main pattern to make this module reusable also to prevent wonky variable scoping

Comment thread GPT2/dataloader.py
Comment on lines +7 to +30
input_file_path = os.path.join(os.path.dirname(__file__), 'input.txt')
if not os.path.exists(input_file_path):
data_url = 'https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt'
with open(input_file_path, 'w', encoding='utf-8') as f:
f.write(requests.get(data_url).text)

with open(input_file_path, 'r', encoding='utf-8') as f:
data = f.read()
n = len(data)
train_data = data[:int(n*0.9)]
val_data = data[int(n*0.9):]

# encode with tiktoken gpt2 bpe
enc = tiktoken.get_encoding("gpt2")
train_ids = enc.encode_ordinary(train_data)
val_ids = enc.encode_ordinary(val_data)
print(f"train has {len(train_ids):,} tokens")
print(f"val has {len(val_ids):,} tokens")

# export to bin files
train_ids = np.array(train_ids, dtype=np.uint16)
val_ids = np.array(val_ids, dtype=np.uint16)
train_ids.tofile(os.path.join(os.path.dirname(__file__), 'train.bin'))
val_ids.tofile(os.path.join(os.path.dirname(__file__), 'val.bin'))

@kamatajinkya2 kamatajinkya2 Jul 8, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
input_file_path = os.path.join(os.path.dirname(__file__), 'input.txt')
if not os.path.exists(input_file_path):
data_url = 'https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt'
with open(input_file_path, 'w', encoding='utf-8') as f:
f.write(requests.get(data_url).text)
with open(input_file_path, 'r', encoding='utf-8') as f:
data = f.read()
n = len(data)
train_data = data[:int(n*0.9)]
val_data = data[int(n*0.9):]
# encode with tiktoken gpt2 bpe
enc = tiktoken.get_encoding("gpt2")
train_ids = enc.encode_ordinary(train_data)
val_ids = enc.encode_ordinary(val_data)
print(f"train has {len(train_ids):,} tokens")
print(f"val has {len(val_ids):,} tokens")
# export to bin files
train_ids = np.array(train_ids, dtype=np.uint16)
val_ids = np.array(val_ids, dtype=np.uint16)
train_ids.tofile(os.path.join(os.path.dirname(__file__), 'train.bin'))
val_ids.tofile(os.path.join(os.path.dirname(__file__), 'val.bin'))
def main():
input_file_path = os.path.join(os.path.dirname(__file__), 'input.txt')
if not os.path.exists(input_file_path):
data_url = 'https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt'
with open(input_file_path, 'w', encoding='utf-8') as f:
f.write(requests.get(data_url).text)
with open(input_file_path, 'r', encoding='utf-8') as f:
data = f.read()
n = len(data)
train_data = data[:int(n*0.9)]
val_data = data[int(n*0.9):]
# encode with tiktoken gpt2 bpe
enc = tiktoken.get_encoding("gpt2")
train_ids = enc.encode_ordinary(train_data)
val_ids = enc.encode_ordinary(val_data)
print(f"train has {len(train_ids):,} tokens")
print(f"val has {len(val_ids):,} tokens")
# export to bin files
train_ids = np.array(train_ids, dtype=np.uint16)
val_ids = np.array(val_ids, dtype=np.uint16)
train_ids.tofile(os.path.join(os.path.dirname(__file__), 'train.bin'))
val_ids.tofile(os.path.join(os.path.dirname(__file__), 'val.bin'))
if __name__ == '__main__':
main()

Comment thread GPT2/train.py
from torch.nn import functional as F
import tiktoken

batch_size = 64

@kamatajinkya2 kamatajinkya2 Jul 9, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dont use global variables

Comment thread GPT2/train.py

return logits,loss


@kamatajinkya2 kamatajinkya2 Jul 9, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use if __name__ == "__main__":

Comment thread GPT2/train.py

def __init__weights(self, module):
if isinstance(module, nn.Linear):
std = 0.02

@kamatajinkya2 kamatajinkya2 Jul 9, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this to else block. This is causing confusion.

Comment thread GPT2/train.cpp
};

struct CausalSelfAttentionImpl : torch::nn::Module {
CausalSelfAttentionImpl(const Config& cfg) {

@kamatajinkya2 kamatajinkya2 Jul 9, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prefer initizliser list. This way your compiler can warn you if there are uninitialized variables

Comment thread GPT2/train.cpp
mask = m;
register_buffer("mask", mask);
}
torch::Tensor forward(const torch::Tensor& x) {

@kamatajinkya2 kamatajinkya2 Jul 9, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread GPT2/train.cpp
auto out = y.permute({0, 2, 1, 3}).contiguous().view({B, T, n_embed});
return proj->forward(out);
}
int64_t n_embed, n_head, head_dim;

@kamatajinkya2 kamatajinkya2 Jul 9, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typically in C++ member variables have a m_ prefix. This prevents variable shadowing

Comment thread GPT2/train.cpp
};
TORCH_MODULE(CausalSelfAttention);

struct MLPImpl : torch::nn::Module {

@kamatajinkya2 kamatajinkya2 Jul 9, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mark this as final so no one accidently inherets.

Comment thread GPT2/train.cpp
TORCH_MODULE(GPT);

int main() {
Config cfg;

@kamatajinkya2 kamatajinkya2 Jul 9, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prefer auto initialization. This prevents uninitialized garbage values

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@kamatajinkya2 kamatajinkya2 kamatajinkya2 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

AltStyle によって変換されたページ (->オリジナル) /