Program to count common words in documents

Question

I finished my first program in Ruby and I would like to share it with you so I can get some suggestions or recommendations. I'd really like to hear them because I am learning and I want to have a solid base. The program has a basic function: it counts the 15 more common words in a document. It is useful in literary studies because it allows the user to know which words is repeating more times, maybe without noticing it. This is how the program works:

The user is able to drag the document into the terminal and to explicitly write the path. Then, the script uses the gem 'yomu' to read it. It reads pdf and docx as well as other files that aren't important for me now. After that, the script makes an array of the words of the document. At this point, the script removes words like prepositions or connectors which aren't important to the user. Finally, it outputs the results in a table through a gem called 'hirb'.

The scripts also offers the possibility to modify the list of words that are going to be removed from the original words. For example, if a document has "Written by Rick Deckard" in every page as a footnote, the user is able to include the words "Written", "by", "Rick" and "Deckard" in the list and they aren't going to be counted. The user can modify the list and save the changes for that session or for ever.

So, there are three scripts: "wordsworth.rb", "core.rb", "forbidden_list_modify_eng.rb". The former script is in the main directory, while the other two are in the path 'languages/english'. That's because I adapted the program to spanish and catalan as well. Anyway, the main script is "wordsworth.rb", which asks the user their language and redirect him or her to the chosen language folder (english, in this case). Once in the english section, the script calls "core.rb", which is the main script of the english section. It means that this scripts is going to do the chief task of the program: count the words and output the results. However, "core.rb" offers the user to modify the list, and this option redirects the user to the third script "forbidden_list_modify_eng.rb" (sorry for the name).

Now I am going to post the three scripts and explain them in detail:

#needs to install 'yomu' and 'hirb' gems
require 'hirb'
require 'yomu'
require 'fileutils'
require_relative "languages/english/forbidden_list_modify_eng.rb"
require_relative "languages/english/core.rb"
def language
 puts "Choose your language:"
 puts "1. Català"
 puts "2. English"
 puts "3. Castellano"
 idiom = gets.chomp.to_s
 case idiom
 when "1", "català", "Català", "CATALÀ"
 system "clear" or system "cls"
 menu_principal_cat
 when "2", "english", "English", "ENGLISH"
 system "clear" or system "cls"
 main_menu
 when "3", "castellano", "Castellano", "CASTELLANO"
 system "clear" or system "cls"
 menu_principal_cast
 else
 system "clear" or system "cls"
 puts "You have to choose a language in the list."
 puts "-----------------------------------------------------"
 language
 end
end
language()

This script is pretty straight-forward, except for the last part, which I'll explain later. It first requires the needed gems and calls the needed scripts. After that, a menu asks the language to the user so it can redirect him or her to the correct language. We suppose the user choose 2 and gets redirected to the function "main_menu", which lands the user to the "core.rb" script:

def tutorial
 puts "The program works like that... (blah blah blah)"
end
def main_menu
 retrieve_forbidden_words = File.read('languages/english/english_list.txt')
 english_list = retrieve_forbidden_words.split("\n")
 puts "Choose what you want to do by writing its number:"
 puts "1: Analyze a document."
 puts "2: Modify the list of forbidden words."
 puts "3: Help me to use this program."
 puts "4: Exit the program."
 main_choice = gets.chomp.to_s
 case main_choice
 when "1"
 system "clear" or system "cls"
 english(english_list)
 when "2"
 system "clear" or system "cls"
 modification_menu
 when "3"
 system "clear" or system "cls"
 tutorial
 when "4"
 exit
 else
 main_menu
 end
end
def english(received_list)
 puts "Drag in here the document you want to analyze:"
 provided_path = gets.chomp.to_s.tr("'", "").rstrip
 execution(provided_path, received_list)
end
def execution (provided_path, forbidden_list)
 data = File.read provided_path
 doc = Yomu.read :text, data
 text_listed = doc.to_s.downcase.gsub(/\p{^Alnum}/, " ").split(" ")
 pruned_text = text_listed - forbidden_list
 frequency = Hash.new 0
 pruned_text.each { |word| frequency[word] += 1 }
 sorted_frequency = Hash[frequency.sort_by{ | word, times | -times }[0..20]]
 toptimes = sorted_frequency.values
 topwords = sorted_frequency.keys
 puts Hirb::Helpers::AutoTable.render(sorted_frequency, headers: {0 => 'Word', 1 => 'Times'}, description: false)
 puts "-----------------------------------------------------"
 main_menu
end

In the "main_menu" the user can choose to analyze the document. If so, the script asks the path of the document in the function "user_doc_path", which takes also the list of forbidden words in English from a .txt file which comes in the program by default. It is a simple file named "english_list.txt" which contains all the words to be removed from the document separated by a line break. So, the user provides the path of the document and the script opens it using 'yum' gem and sort it to get the words in an array. After that, the script creates a hash that stores the number of times each word is repeated throughout the text. Then, the program orders the hash, reverses it and get the 15 top results which finally output in a table provided by 'hirb' gem.

If the user choose to modify the list of words that are removed from the document, he or she is redirected to the "forbidden_list_eng_modify.rb":

def remove_words(list_to_remove)
 puts "Write the words you want to remove from the list. If there are more than one, divide them by commas."
 remove_propose = gets.chomp.tr(" ", "").to_s.split(",")
 new_list = list_to_remove - remove_propose
 File.new("languages/english/english_backup.txt", "w")
 File.open("languages/english/english_backup.txt", "w+") do |f|
 f.puts(list_to_remove)
 end
 File.open("languages/english/english_list.txt", "w+") do |f|
 f.puts(new_list)
 end
 puts "The words have been removed. Do you want to see the new list? [Yes/No]"
 showit = gets.chomp.tr(" ", "").to_s
 case showit
 when "Yes", "yes", "Y", "y"
 system "clear" or system "cls"
 print new_list
 puts "-----------------------------------------------------"
 modification_menu
 when "No", "no", "N", "n"
 system "clear" or system "cls"
 modification_menu
 end
end
def add_words(list_to_add)
 puts "Write the words you want to add to the list. If there are more than one, divide them by commas."
 add_propose = gets.chomp.tr(" ", "").to_s.split(",")
 new_list = list_to_add + add_propose
 File.new("languages/english/english_backup.txt", "w")
 File.open("languages/english/english_backup.txt", "w+") do |f|
 f.puts(list_to_add)
 end
 File.open("languages/english/english_list.txt", "w+") do |f|
 f.puts(new_list)
 end
 puts "The words have been added. Do you want to see the new list? [Yes/No]"
 showit = gets.chomp.tr(" ", "").to_s
 case showit
 when "Yes", "yes", "Y", "y"
 system "clear" or system "cls"
 print new_list
 puts "-----------------------------------------------------"
 modification_menu
 when "No", "no", "N", "n"
 puts "-----------------------------------------------------"
 modification_menu
 end
end
def modification_menu
retrieve_forbidden_words = File.read('languages/english/english_list.txt')
original_list = retrieve_forbidden_words.split("\n")
puts "This list contains all the words that are automatically removed from the document you want to analyze. The words in it are widely-used words with no special meaning like connectors or prepositions. However, you can modify the list as you want."
puts "1: Show all the words of the list."
puts "2: Remove words."
puts "3: Add words."
puts "4: Save the new list for other sessions."
puts "5: Return to the main menu."
list_choice = gets.chomp.to_s
case list_choice
when "1"
 system "clear" or system "cls"
 print original_list
 puts ""
 puts "-----------------------------------------------------"
 modification_menu
when "2"
 system "clear" or system "cls"
 remove_words(original_list)
when "3"
 system "clear" or system "cls"
 add_words(original_list)
when "4"
 backup_existence = File.file?('languages/english/english_backup.txt')
 if backup_existence == TRUE
 File.delete('languages/english/english_backup.txt')
 system "clear" or system "cls"
 puts "Changes has been succesfully saved."
 puts "-----------------------------------------------------"
 modification_menu
 end
 if backup_existence == FALSE
 system "clear" or system "cls"
 puts "The list hasn't been modified, so it's nothing to save."
 puts "-----------------------------------------------------"
 modification_menu
 end
when "5"
 system "clear" or system "cls"
 main_menu
else
 system "clear" or system "cls"
 modification_menu
end
end

This script shows the user a menu which offers the possibility of showing the list, adding or removing some words and saving the changes for ever. The first option, show the complete list, just prints the output of the "english_list.txt", which is the file where the words are stored. The "add_words", as the "remove_words" function, asks the user to write the words that want to get removed separating them by commas. The script just removes the provided array from the default list. At this point, the script creates a "english_backup.txt" which is the back up in case the user closes the program without saving. The modified list is named "english_list.txt", and it works as the new default list, so the user can analyze more texts in the same session with that list. If the user chooses to save the changes, the "english_backup.txt" is destroyed and the new "english_list.txt" is the main list. If the user exits, the "english_backup.txt" will get its former name ("english_list.txt") and the unsaved list file will be removed. This is achieved in the former script, "wordsworth.rb", when END is detected it proceeds to remove the unsaved file and restore the backup as main list.

• thesecretmaster: Just wanted to let you know that I really like the question, and I've been working on a review. I'll post it (and delete this comment) at some point in the next few days.
• H4ml3tt3d: Hello! That makes me really happy! I thought the question was forgotten in the stackexhange unstoppable flow. I'm looking forward to reading your review :)

Answer 1

This is a really interesting project, and looks pretty solid. I'm just going to go through and give you some ruby style tips as I see them. I've annotated all of the files with my thoughts.

First of all, you use lists in a couple places. Because of that, I'd suggest that you create a generic list method right above your #language method:

def list(*options, retry_msg: '')
 options = Array(options).flatten.map(&:to_s).uniq
 invalid_opts = (0..options.length).map(&:to_s)
 throw "You cannot pass a number as an option" if options.any? { |opt| invalid_opts.include?(opt) }
 loop do
 puts options.map.with_index { |opt, i| "#{i}. #{opt}" }
 selection = gets.chomp
 system "clear" or system "cls"
 if (0..options.length-1).map(&:to_s).include? selection
 return options[selection.to_i]
 else
 selection = options.select { |opt| opt.downcase == selection.downcase }.first
 return selection unless selection.nil?
 end
 puts retry_msg
 end
end

Side note: Since this would remove your lists throughout, I just wanted to add some code style tips in case you decided to leave the list in. Here are my comments on the list in your #language method:

def language
 # I would simplify the listing of languages (and make it more extensible)
 # by having an array of languages you support.
 puts "Choose your language:"
 # I've created the array using the %w literal, which creates an array
 # of words split on the spaces.
 languages = %w[Català English Castellano]
 # puts "1. Català"
 # puts "2. English"
 # puts "3. Castellano"
 # And then print them out: (puts automatically adds newlines
 # between the array elements)
 puts languages.map.with_index { |lang, index| "#{index}. #{lang}" } 
 idiom = gets.chomp.to_s
 # Here, you can convert the idiom to lowercase so you don't need
 # to have different capitalizations in your when statements
 case idiom.downcase
 when "1", "català"#, "Català", "CATALÀ"
 system "clear" or system "cls"
 menu_principal_cat
 when "2", "english"#, "English", "ENGLISH"
 system "clear" or system "cls"
 main_menu
 when "3", "castellano"#, "Castellano", "CASTELLANO"
 system "clear" or system "cls"
 menu_principal_cast
 else
 system "clear" or system "cls"
 puts "You have to choose a language in the list."
 puts '-'*53 # You can use multiplication instead of typing the char so many times
 language
 end
end

Anyways, here's an actual review of your main file:

require 'hirb'
require 'yomu'
require 'fileutils'
require_relative "languages/english/forbidden_list_modify_eng.rb"
require_relative "languages/english/core.rb"
# The addition of the list method means that all this code doesn't need
# to be wrapped in a language method
# def language
 puts "Choose your language:"
 # puts "1. Català"
 # puts "2. English"
 # puts "3. Castellano"
 # idiom = gets.chomp.to_s
 retry_msg = "You have to choose a language in the list.\n"
 retry_msg += '-'*53
 case list(%w[Català English Castellano], retry_msg: retry_msg)
 when "Català"
 menu_principal_cat
 when "English"
 main_menu
 when "Castellano"
 menu_principal_cast
 end
# end
# language()

And, for the second file:

def tutorial
 # If this is really long, it might be a good idea to put
 # the contents of the tutorial in a text file and instead
 # use puts File.read("path/to/file")
 puts "The program works like that... (blah blah blah)"
end
def main_menu
 # I've renamed the variable to be forbidden_words (from retrieve_forbidden_words) because the variable is the forbidden words, not the action of retrieving them.
 forbidden_words = File.read('languages/english/english_list.txt')
 english_list = forbidden_words.split("\n")
 # Here we can use the magic list method
 puts "Choose what you want to do by writing its number:"
 # puts "1: Analyze a document."
 # puts "2: Modify the list of forbidden words."
 # puts "3: Help me to use this program."
 # puts "4: Exit the program."
 # main_choice = gets.chomp.to_s
 # case main_choice
 options = []
 options << 'Analyze a document.'
 options << 'Modify the list of forbidden words.'
 options << 'Help me to use this program.'
 options << 'Exit the program.'
 case list(options)
 when 'Analyze a document.'
 english(english_list)
 when 'Modify the list of forbidden words.'
 modification_menu
 when 'Help me to use this program.'
 tutorial
 when 'Exit the program.'
 exit
 end
end
def english(received_list)
 puts "Drag in here the document you want to analyze:"
 provided_path = gets.chomp.to_s.tr("'", "").rstrip
 execution(provided_path, received_list)
end
def execution (provided_path, forbidden_list)
 data = File.read provided_path
 doc = Yomu.read :text, data
 text_listed = doc.to_s.downcase.gsub(/\p{^Alnum}/, " ").split(" ")
 pruned_text = text_listed - forbidden_list
 frequency = Hash.new 0
 pruned_text.each { |word| frequency[word] += 1 }
 sorted_frequency = Hash[frequency.sort_by{ | word, times | -times }[0..20]]
 toptimes = sorted_frequency.values
 topwords = sorted_frequency.keys
 puts Hirb::Helpers::AutoTable.render(sorted_frequency, headers: {0 => 'Word', 1 => 'Times'}, description: false)
 puts "-----------------------------------------------------"
 main_menu
end

And, the final file:

def remove_words(list_to_remove)
 puts "Write the words you want to remove from the list. If there are more than one, divide them by commas."
 remove_propose = gets.chomp.tr(" ", "").to_s.split(",")
 new_list = list_to_remove - remove_propose
 File.new("languages/english/english_backup.txt", "w")
 File.open("languages/english/english_backup.txt", "w+") do |f|
 f.puts(list_to_remove)
 end
 File.open("languages/english/english_list.txt", "w+") do |f|
 f.puts(new_list)
 end
 puts "The words have been removed. Do you want to see the new list? [Yes/No]"
 showit = gets.chomp.tr(" ", "").to_s
 case showit
 when "Yes", "yes", "Y", "y"
 system "clear" or system "cls"
 print new_list
 puts "-----------------------------------------------------"
 modification_menu
 when "No", "no", "N", "n"
 system "clear" or system "cls"
 modification_menu
 end
end
def add_words(list_to_add)
 puts "Write the words you want to add to the list. If there are more than one, divide them by commas."
 add_propose = gets.chomp.tr(" ", "").to_s.split(",")
 new_list = list_to_add + add_propose
 File.new("languages/english/english_backup.txt", "w")
 File.open("languages/english/english_backup.txt", "w+") do |f|
 f.puts(list_to_add)
 end
 File.open("languages/english/english_list.txt", "w+") do |f|
 f.puts(new_list)
 end
 puts "The words have been added. Do you want to see the new list? [Yes/No]"
 showit = gets.chomp.tr(" ", "").to_s
 case showit
 when "Yes", "yes", "Y", "y"
 system "clear" or system "cls"
 print new_list
 puts "-----------------------------------------------------"
 modification_menu
 when "No", "no", "N", "n"
 puts "-----------------------------------------------------"
 modification_menu
 end
end
def modification_menu
retrieve_forbidden_words = File.read('languages/english/english_list.txt')
original_list = retrieve_forbidden_words.split("\n")
puts "This list contains all the words that are automatically removed from the document you want to analyze. The words in it are widely-used words with no special meaning like connectors or prepositions. However, you can modify the list as you want."
puts "1: Show all the words of the list."
puts "2: Remove words."
puts "3: Add words."
puts "4: Save the new list for other sessions."
puts "5: Return to the main menu."
list_choice = gets.chomp.to_s
case list_choice
when "1"
 system "clear" or system "cls"
 print original_list
 puts ""
 puts "-----------------------------------------------------"
 modification_menu
when "2"
 system "clear" or system "cls"
 remove_words(original_list)
when "3"
 system "clear" or system "cls"
 add_words(original_list)
when "4"
 backup_existence = File.file?('languages/english/english_backup.txt')
 if backup_existence == TRUE
 File.delete('languages/english/english_backup.txt')
 system "clear" or system "cls"
 puts "Changes has been succesfully saved."
 puts "-----------------------------------------------------"
 modification_menu
 end
 if backup_existence == FALSE
 system "clear" or system "cls"
 puts "The list hasn't been modified, so it's nothing to save."
 puts "-----------------------------------------------------"
 modification_menu
 end
when "5"
 system "clear" or system "cls"
 main_menu
else
 system "clear" or system "cls"
 modification_menu
end
end

---

Side note: My apologies for taking so long to get back to this, I just got busy IRL and couldn't finish.