Chris Lomont's lines of code over time

Published 2019年06月23日

Over decades I’ve written a lot of personal code. I made a bar chart race showing how many lines I’ve written in different languages over time.

Here is a bar chart race for code I’ve written over the decades.

Method

I took every source file on my hard drive, did cleanup to remove duplicates and those not mine and so on, counted lines in each file and noted the creation date, and visualized the data in a bar chart race.

I wrote a script (below) in F# to analyze the data and put it in a format suitable for an online bar chart race generator at https://app.flourish.studio/@flourish/bar-chart-race.

(I did start making my own bar chart race generator to give me more control but decided to stop spending too much time on this.)

Details

Here are the steps my program took to gather statistics

I selected all filenames with specific extensions that were sub to certain directories.
To try and count legitimate lines of code, I did several passes of cleanup. I removed expected duplicates via MD5 hashing, and I did several visual scans through all files and added patterns to exclude. This could be strengthened to count unique lines to help detect derivative files.
Sorted by date.
Created running tallies by file extension.
Export as CSV in format suitable for the bar chart race generator.

Sources of error

As a kid I wrote a lot of assembly code on a TRS-80 Color Computer, but those files are offline (I still have them on disk and cassette, if I ever get around to reading them out). So this animation starts after that era.

I possibly missed other directories - in the process of doing this I found a few places I had tucked away old code.

I may have some eras in zip files.

I originally added Mathematica *.nb files, but they were too large, since they have code and generated data intermixed. I have written a significant amount of them, but I threw them out.

I did add *.bat files since I wrote a lot of them, but there are also many I didn’t counted.

I ignored Powershell scripts *.ps since I have too many postscript files around.

I suspect there is a lot of overcounting from derived code files.

This also doesn’t have most if any of my professional code, since I don’t have it to analyze.

Description of visualization

Around 1988 or so I got a PC, and I started writing code in Turbo Pascal. My senior project in undergrad in 1991 was a chess program, which I still have. I also wrote a lot of assembly code for fun and learning.

In 1992 I worked in Chicago, and I bought Turbo C++ to learn C and C++, so the code turns to them along with the assembly code.

I wrote games and did game consulting in the mid 90s, and then went to grad school. All my side work was still C/C++/asm, with a tiny bit of Python.

In the mid 90s through early 2000s, I was in grad school, and I wrote a lot of code for fun.

In the mid-late 2000s, I started doing my side code in C#, which catches up pretty quickly.

In the mid 2010s, I worked fulltime on my own projects, so my professional code count shows up as well.

Code

Here is the F# I used. To use it, scan the top and replace the file types, the exclusion lists, and whatever else needs cleaned for your needs. I’m posting it in a messy state, but it worked so far.

I do plan on cleaning it up and this post to get more accurate counts.

// Learn more about F# at http://fsharp.org
open System
open System.Text
open System.IO
open System.Collections.Generic
// open Opulos.Core.IO
open System.Diagnostics
open System.Security.Cryptography
(* Todo
- Exclusions by list
- Detect dups and dont count
- 
*)
(*
Things to gather
- list of all
 (filename, extension, year, month, day, day of week, linecount)
 or possibly (if not then recomputing fields many times)
 (filename,Date,linecount)
- all extensions and counts
- exclude things TODO
- all list of
 (filename, extension, year,month, day, day of week, linecount)
 -- for each month, count total lines and lines by filetype
 -- store as map[year*100+month,map[filetype,int]]
Nice graphs:
- code lines by month broken down into type and colored
- code total lines in pie by type
- code by day of week
- # code files of type lines per file over time
- # projects - each in own dir?
- make bar chart race
- total pic: 
 use SkiaSharp, Bar chart race in UL corner, total growth LL with progress, 
 Per month (or week) in UR, and about in LR
*)
// code extensions we care about
let sourceExtensions = [
 ".cxx";".c";".cpp";".h";".hpp";
 ".cs";".fs";".xaml"; 
 ".asm";".s";".py";".js";".go"; ".pas";
 ".rc";
 ".bat"; // too much noise
 ".m"; 
 // ".ma"; ".nb"; // far too many lines, generated
 "bas"
 ]
// sequence to lowercase
// replace '\' with '/'
let normalize (str:string list) = 
 [ for s in str do 
 let s1 = s.Replace("\\","/")
 yield s1.ToLower() 
 ]
// exclusions from above
let excludeSuffixes = ["g.i.cs"]
let excludeContains = normalize [
 "/boost/";
 "/boost_1_28_0/";
 "/boost_old/";
 "/buildprocesstemplates/";
 "/libjpeg/";
 "/libpng/";
 "/loki/";
 "/opengl/";
 "/zlib/";
 "/extfilesystem/";
 "/lemmings.net/";
 "/properties/";
 "/xsharp/";
 "/wolfsrc/src/";
 "/antigraingeometry/";
 "/keccakreferenceandoptimized/";
 "/golly-1.1-src/";
 "/poker-eval-129.0/";
 "/id3lib/";
 "/lua/";
 "/borlandc/";
 "/at2g3r38sourcecode/";
 "/fmt-master/";
 "/stockfish/";
 "/relacy/";
 "/stdmarkdown/";
 "/opengl line rendering/code/";
 "/pmars-0.9.2/";
 "/obj/";
 "/debug/";
 "/release/";
 "/properties/";
 "/prettycomments_integrationtests/";
 "/zlibnet/";
 // new
 "/zlib.net/";
 "/sourcesafedata/";
 "/winwolf3d/";
 "/fontmaker/bitmapfonts/";
 "/usbgadgetdriver/usb/";
 "/libf2c/";
 "/rfvdemo/";
 "/ez_lcd_wrapper/";
 "/liblz4/";
 "/google job interview/reservations_files/";
 "/lemc/arm9/";
 "/lemmingsds/lemmingsvc/";
 "/golly/";
 "/skein/nist/";
 "/externals/";
 ".acceptancetests/";
 "/adplug";
 "/dosbox";
 "/libxmp";
 "/relacy";
 "/gumstix/";
 "/spiffs/";
 "/meyers/";
 "/applewin/";
 "/tundra python";
 "/massivelyparallelvectorgraphics/";
 "/turbo55/oop_demo/";
 "/antlr";
 "/visualgdbcache/";
 "/packages/";
 "/include/config/";
 "/c/wolfsrc/";
 "/icsharpcode.texteditor/";
 "/amber";
 "/dr dobbs";
 "/myers/";
 "/cstuff/gems";
 "/cstuff/wolfsrc/";
 "/me/src/";
 "/sound/modplay/";
 "/sound/sbdox/";
 "/sound/voc/";
 "/gadgetdriverold/";
 "/e_dec96/graphics/src/";
 "/zork123/";
 "/tume/";
 "/fof/";
 "/png/";
 "/plytool/";
 "/shellcrypt/";
 "/adplay/";
 "/turboprg/";
 ]
// file paths to check
//let dirs = [ "D:\Programming\Code2"]
let dirs = normalize [ 
 "D:\Programming\Code2"; 
 "D:\Programming\CSTUFF2"; 
 "C:\Users\Chris\OneDrive\Code"; 
 "C:\Users\Chris\Desktop"; 
 "D:\Hypnocube\Development\Code";
 "D:\CCLText"
 ]
//let dirs = [ "D:\Programming\Code2\FlashAnalyzer" ; "C:\Users\Chris\OneDrive\Flash Code" ]
//let dirs = ["C:\Users\clomont\Documents\Code\DataScience";"C:\Users\clomont\Documents\Code\Math3D"]
//let dirs = ["C:\Users\clomont\Documents\Code\Qt"]
//let dirs = ["C:\Users\clomont\Desktop\CSTUFF2"]
// file sequence to extension
let extension (files:string list) = [ for f in files do yield Path.GetExtension f ]
// see if item has a suffix from a list
let hasSuffix (item:string) (suffixes: string list) = 
 List.exists (fun (suffix:string) -> item.EndsWith(suffix)) suffixes
let containsAny (item:string) (matches:string list) = 
 matches |> List.exists (fun m -> item.Contains(m))
// let only code through this filter that has no matched patterns
let codeOnly (files : string list) = 
 files 
 |> normalize 
 |> Seq.filter(fun fn -> List.contains (Path.GetExtension(fn)) sourceExtensions )
 |> Seq.filter(fun fn -> not (hasSuffix fn excludeSuffixes))
 |> Seq.filter(fun fn -> not (containsAny fn excludeContains))
 |> Seq.toList
// turn a sequence of directory names into files
let rec allFiles (dirs : string list) =
 if List.isEmpty dirs then List.empty else
 [ yield! dirs |> Seq.collect Directory.EnumerateFiles
 yield! dirs |> Seq.collect Directory.EnumerateDirectories |> Seq.toList |> allFiles ]
// DateTime GetLastFileModifiedFast(string dir)
// {
// DateTime retval = DateTime.MinValue;
// 
// FileData [] files = FastDirectoryEnumerator.GetFiles(dir);
// for (int i=0; i<files.Length; i++)
// {
// if (files[i].LastWriteTime > retval)
// {
// retval = lastWriteTime;
// }
// }
// 
// return retval;
// }
//let allFilesFast dirs = 
// FastFileInfo.EnumerateFiles(Seq.head dirs, "*", SearchOption.AllDirectories)
// if Seq.isEmpty dirs then Seq.empty else
// seq { 
// for d in dirs do
// yield! FastFileInfo.EnumerateFiles(d, "*", SearchOption.AllDirectories)
// }
 //FastFileInfo.EnumerateFiles(path, "*.*", SearchOption.AllDirectories)
 //FastDirectoryEnumerator.GetFiles
// get lowercase extensions and count them
// return sequence of string*int
let tallyLowercaseExtensions (files : string list) = files |> List.countBy (fun fn->Path.GetExtension fn) 
// length of file
let countLines filename = File.ReadAllLines(filename).Length
// print all items in sequence
let printSeq s = 
 let rec phelp s c =
 if Seq.isEmpty s then c
 else
 printfn "%A" (Seq.head s)
 phelp (Seq.tail s) (c+1)
 phelp s 0
(*********************************** various code sampling things **********************)
// list all files to output, return count
let listFiles files = 
 let count = printSeq files
 printfn "Count %A" count
 count
// tally items and output
let listTally s = 
 let fastTally s : ((string*int) list) = 
 let d = new Dictionary<string,int>()
 s |> Seq.iter(fun t -> 
 if d.ContainsKey(t) then ignore()
 else d.Add(t,0)
 d.[t] <- d.[t] + 1
 )
 [ for (KeyValue(k,v)) in d do yield (k,v) ] |> List.sortBy (fun (k,v)-> (-v))
 // let t = s |> Seq.countBy (fun f->f) 
 let t = fastTally s
 let count = printSeq t
 printfn "Count %A" count
 count
// one record per file
type Record = 
 {
 filename : string
 extension : string
 date: DateTime
 hash : string
 //year : int
 //month : int
 //day : int
 //hour : int
 //weekday : DayOfWeek
 linecount : int
 }
type TimeStep = 
 | Daily
 | Weekly
 | Biweekly
 | Monthly
 | Bimonthly
 | Quarterly
let hashFile (md5:MD5) f = 
 let bytes = File.ReadAllBytes f
 (StringBuilder(), md5.ComputeHash(bytes)) ||> Array.fold (fun sb b -> sb.Append(b.ToString("x2"))) |> string
// convert files to seq of records
let toRecords s = 
 let md5 = MD5.Create()
 s |> List.map(fun f ->
 let date = File.GetLastWriteTime f
 let ext = Path.GetExtension f
 let hash = hashFile md5 f
 {filename = f; extension = ext; hash = hash; date = date; linecount = (countLines f) }
 )
// allow date time iteration
type Span = Span of TimeSpan with
 static member (+) (d:DateTime, Span wrapper) = d + wrapper
 static member Zero = Span(new TimeSpan(0L))
// count lines by month, return count of all lines
// takes a sequence of all records
let computeLines recs timestep = 
// // year and month encoded as 100*y+m
// let encode (r:Record) = 
// match timestep with
// | Month -> 100*(100*r.year + r.month) + 15 // all on mid month
// | Week -> 100*(r.year + r.month) + (r.day/7)*7
// | Day -> 100*(r.year + r.month) + r.day
// 
// let decode key = 
// let y = key/10000
// let m = (key%10000)/100
// let d = key%100
// (y,m,d)
//
// printfn "Do min max"
 // get min and max
 //let rec minmax r (min,max) = 
 // if Seq.isEmpty r then (min,max)
 // else
 // let v = encode (Seq.head r)
 // minmax (Seq.tail r) (Math.Min(min,v), Math.Max(max,v))
 //let (min,max) = minmax recs (Int32.MaxValue, Int32.MinValue)
 // sorted file types that occur
 let types = recs |> List.map (fun r->r.extension) |> Set.ofList |> Set.toList |> List.sort
 let typeCount = List.length types
 printfn "Types %A" types
 // map from code type to name index
 let typemap = types |> List.mapi (fun i x -> x,i) |> Map.ofList
 // snap date to nearby value
 let snapDate (date:DateTime) = 
 let y,m,d = date.Year, date.Month, date.Day
 match timestep with 
 | Daily -> new DateTime(y,m,d)
 | Weekly -> 
 // 1,8,15,22 
 let d2 = ((d-1)/7) * 7 + 1
 let d3 = if d2 > 22 then 22 else d2
 new DateTime(y,m,d3)
 | Biweekly ->
 // 1,15
 let d2 = ((d-1)/14) * 14 + 1
 let d3 = if d2 > 15 then 15 else d2
 new DateTime(y,m,d3)
 | Monthly -> 
 new DateTime(y,m,1)
 | Bimonthly ->
 // m = 1,3,5,7,11
 let m2 = ((m-1)/2)*2+1
 new DateTime(y,m2,1)
 | Quarterly ->
 // m = 1,4,7,10
 let m2 = ((m-1)/3)*3+1
 new DateTime(y,m2,1)
 // find first date
 let dates = recs |> Seq.map(fun r->r.date)
 let startDate = dates |> Seq.min
 let endDate = dates |> Seq.max
 printfn "Date range %A to %A" startDate endDate
 printfn "Do array"
 let dayData = new Dictionary<DateTime,(int array)>()
 // populate
 let days = 
 seq {
 let mutable d = snapDate startDate
 let e = snapDate endDate
 while d < e do 
 yield d
 d <- d + TimeSpan.FromDays(1.0)
 yield e
 } |> Seq.toList
 printfn "Column count %A" days.Length
 for d in days do
 let snap = snapDate d
 if not (dayData.ContainsKey(snap)) then
 dayData.Add(snap, Array.create (types.Length) 0)
 printfn "Num entries %A" dayData.Keys.Count
 // add record to data
 let addTo (r:Record) = 
 let date = snapDate r.date
 let m = dayData.[date]
 let index = typemap.[r.extension]
 m.[index] <- m.[index] + r.linecount
 printfn "Do fill"
 for item in recs do
 addTo item
 // sorted data
 printfn "Do sort"
 let array = dayData |> Seq.sortBy(fun (KeyValue(k,v))->k) |> Seq.map(fun (KeyValue(k,v))->(k,v))
 printfn "Do output"
 // now output 
 use file = File.CreateText("Data.csv")
 // header and dates
 file.Write("Name,")
 array |> Seq.iter (fun (date,_) -> 
 let y,m,d = date.Year,date.Month,date.Day
 file.Write(String.Format("{0}/{1}/{2},",y,m,d))
 )
 file.WriteLine()
 for i in [0..types.Length-1] do
 let t = types.[i]
 file.Write(t+",")
 let mutable sum = 0
 array |> Seq.iter (fun (date,data) -> 
 let v = data.[i]
 sum <- sum + v
 file.Write(String.Format("{0},",sum))
 )
 file.WriteLine()
// array |> Array.iteri (fun i (total,arr) -> 
// let y,m,d = decode (i+min)
//
// printf "%A-%A-%A %A: " y m d total
// arr |> Array.iteri (fun j count ->
// printf "%A=%A" types.[j] count
// )
// printfn ","
// )
 file.Close()
 printfn "File written"
let timer = new Stopwatch();
let timeMsg msg = 
 printf msg
 printfn ": %A" timer.Elapsed
let time f = 
 let proc = Process.GetCurrentProcess()
 let cpu_time_stamp = proc.TotalProcessorTime
 let timer = new Stopwatch()
 timer.Start()
 try
 f()
 finally
 let cpu_time = (proc.TotalProcessorTime-cpu_time_stamp).TotalMilliseconds
 printfn "CPU time = %dms" (int64 cpu_time)
 printfn "Absolute time = %dms" timer.ElapsedMilliseconds
let rec loop n f x =
 if n > 0 then
 f x |> ignore
 loop (n-1) f x
let runit() = 
 timer.Start()
 printfn "Do files"
 let files = (allFiles dirs) |> Seq.toList
 timeMsg "files done"
 //printfn "Do files 2"
 //let f2 = allFilesFast dirs |> Seq.toList
 //timeMsg "files 2 done"
 //printfn "Sizes %A %A" files.Length f2.Length
 printfn "Do derived"
 let lowercaseFiles = files |> normalize 
 let lowercaseCodeFiles = lowercaseFiles |> codeOnly
 printfn "%A code files " lowercaseCodeFiles.Length
 let removed = lowercaseCodeFiles |> List.filter (fun fn -> containsAny fn excludeContains)
 for r in removed do 
 printfn "%A" r
 //ignore <| listFiles files 
 //ignore <| listFiles lowercaseFiles
 //ignore <| listFiles lowercaseCodeFiles
 // extension count of all 
 //ignore <| listTally (lowercaseFiles |> extension)
 // extension count of code
 //ignore <| listTally (lowercaseCodeFiles |> extension)
 printfn "Do recs"
 // (filename, extension, year, month, day, day of week, linecount)
 let mutable recs = lowercaseCodeFiles |> toRecords |> List.filter(fun r -> r.date.Year > 1989) |> List.sortBy(fun r -> r.date)
 printfn "Rec from %A to %A " recs.[0].date recs.[recs.Length-1].date
 //for i in [0..10] do
 // printfn "%A" recs.[i]
 let linesOfCode = recs |> List.sumBy(fun r-> r.linecount)
 printfn "Lines of code %A" linesOfCode
 // remove dups, dump again
 printfn "Removing dups"
 let mutable s = Set.empty
 let crecs = seq {
 for r in recs do
 if not (s.Contains r.hash) then
 yield r
 s <- s.Add(r.hash)
 } 
 recs <- crecs |> Seq.toList
 printfn "Lines of code %A" (recs |> List.sumBy(fun r->r.linecount))
 // dump files
 use f = File.CreateText("Files.txt")
 for c in recs do
 f.WriteLine(c.filename)
 f.Close()
 printfn "Files dumped"
 // ignore <| printSeq dat
 printfn "Do lines"
 // lines by month or day
 computeLines recs Bimonthly
 0 // return an integer exit code
let testit() = 
 ignore()
 let mutable c = 0
 let files = 
 seq {
 for f in File.ReadAllLines("exclude1Files.txt") do
 yield f
 } |> Seq.filter (fun f -> not (String.IsNullOrEmpty f))
 |> Seq.toList
 |> codeOnly
 printfn "Test files %A" files.Length
 for c in files do
 printfn "%A" c
// let test = 
// [
// //"d:/programming/code2/buildprocesstemplates/defaulttemplate.11.1.xaml";
// "d:/programming/cstuff2/boost_1_28_0/boost/any.hpp"
// ] |> codeOnly 
// ignore <| listFiles test
[<EntryPoint>]
let main argv =
 //testit()
 runit()
 0 // return an integer exit code

Categories: Software

Tags: Software F# C# C++ Languages

Comments

Comment is disabled to avoid unwanted discussions from 'localhost:1313' on your Disqus account...