HTML downloader and parser for CR

Question

This program downloads a Code Review HTML file and parses it.

Could you review my program?

Main.java

import java.net.URL;
public class Main {
 public static void main(String[] args) throws Exception {
 final String site1 = "http://codereview.stackexchange.com/questions/";
 String site2;
 String site3;
 URL url;
 HtmlGetter htmlGetter = new HtmlGetter();
 while(true) {
 site2 = "69";
 site3 = "is-this-implementation-of-shamos-hoey-algorithm-ok";
 url = new URL(site1 + site2 + "/" + site3);
 htmlGetter.setFileName("[" + site2 + "]" + site3 + ".html");
 htmlGetter.download(url);
 htmlGetter.parse();
 break;
 }
 }
}

HtmlGetter.java

import java.io.File;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.net.URL;
import java.util.Scanner;
import org.apache.commons.lang3.StringEscapeUtils; 
import org.apache.commons.io.IOUtils;
public class HtmlGetter {
 private String fileName;
 public void setFileName(String fileName) {
 this.fileName = fileName;
 }
 public String getFileName() {
 return fileName;
 }
 public void download(URL url) throws Exception {
 final InputStream in = url.openStream();
 final OutputStream out = new FileOutputStream(fileName);
 IOUtils.copy(in, out);
 in.close();
 out.close();
 }
 public void parse() throws Exception {
 Scanner scanner = new Scanner(new File(fileName));
 String oneLine;
 String htmlTagRegex = "<(/)?([a-zA-Z]*)(\\s[a-zA-Z]*=[^>]*)?(\\s)*(/)?>";
 while(scanner.hasNextLine()) {
 oneLine = scanner.nextLine();
 if(oneLine.matches(".*<code>.*")) {
 while(scanner.hasNextLine()&&(!oneLine.matches(".*</code>.*"))) {
 oneLine = StringEscapeUtils.unescapeHtml4(oneLine.replaceAll(htmlTagRegex, ""));
 System.out.println(oneLine);
 oneLine = scanner.nextLine();
 // This cannot handle such kind of a line which has <code> and </code> together.
 }
 System.out.println("-------------------------------------------");
 }
 } 
 }
}

Answer 1

public static void main(String[] args) throws Exception {
 final String site1 = "http://codereview.stackexchange.com/questions/";
 String site2;
 String site3;

Why do define site1 as a final constant, but delay the other assignments until later? They should really be static finals for the class. Also some clearer names would not go amiss.

 URL url;

Declare them where they are used, don't declare them ahead of time.

 HtmlGetter htmlGetter = new HtmlGetter();
 while(true) {
 site2 = "69";
 site3 = "is-this-implementation-of-shamos-hoey-algorithm-ok";
 url = new URL(site1 + site2 + "/" + site3);
 htmlGetter.setFileName("[" + site2 + "]" + site3 + ".html");
 htmlGetter.download(url);
 htmlGetter.parse();
 break;

Why do you have a loop if you are just going to break out of it? } }

public void parse() throws Exception {

You shouldn't throw Exception, your specification should really only include the exceptions that'll actually be thrown

 Scanner scanner = new Scanner(new File(fileName));
 String oneLine;
 String htmlTagRegex = "<(/)?([a-zA-Z]*)(\\s[a-zA-Z]*=[^>]*)?(\\s)*(/)?>";

Parsing HTML by regex is asking for trouble. Instead, you'll find the task much easier if you grab a html parsing library and use that.

 while(scanner.hasNextLine()) {
 oneLine = scanner.nextLine();
 if(oneLine.matches(".*<code>.*")) {

You are just searching for a substring here, there isn't much point in using regex to do that.

 while(scanner.hasNextLine()&&(!oneLine.matches(".*</code>.*"))) {

If you insist on regex, use a single regex to capture the entire code tag.

 oneLine = StringEscapeUtils.unescapeHtml4(oneLine.replaceAll(htmlTagRegex, ""));
 System.out.println(oneLine);
 oneLine = scanner.nextLine();
 // This cannot handle such kind of a line which has <code> and </code> together.
 }
 System.out.println("-------------------------------------------");
 }
 } 
}

Using something like jsoup: http://jsoup.org/

Document question = Jsoup.connect("http://codereview.stackexchange.com/questions/69").get();
Elements codes = question.select("code")
for(Element code: codes)
{
 System.out.println(code.text())
}