Return to Answer

Some of the general things I would work on:

• split the code into separate modules logically. Currently, you have all the code mixed up in single file - you have argument parsing, database interactions, web-scraping code blocks in one place
• consistent indentation. Use 4 spaces for indentation
• variable naming. Use descriptive variable names. Variable names like l, i or j are not meaningful and raise questions when reading the code

###Code Style

• avoid handling broad exceptions with a bare except
• remove unused imports and re-group them based on PEP8 recommendations
• make sure to properly use whitespaces in expressions in statements
• put the main execution logic of the program into the if __name__ == '__main__':
• you don't need that pass in the get_list_of_urls() function
• depth = depth + 1 could be shortened to depth += 1

###Performance

• since you are requesting the pages from the same host multiple times, consider switching to requests making use of a single session instance which allows to re-use an underlying TCP connection making subsequent requests to the same host faster
• importing SoupStrainer was actually a good idea. You can use it and scope the parsing to only the desired parts of the HTML
• since you are using regular expressions checks here and there, consider pre-compiling them and using the compiled patterns for searching and matching

Some of the general things I would work on:

• split the code into separate modules logically. Currently, you have all the code mixed up in single file - you have argument parsing, database interactions, web-scraping code blocks in one place
• consistent indentation. Use 4 spaces for indentation
• variable naming. Use descriptive variable names. Variable names like l, i or j are not meaningful and raise questions when reading the code

Code Style

• avoid handling broad exceptions with a bare except
• remove unused imports and re-group them based on PEP8 recommendations
• make sure to properly use whitespaces in expressions in statements
• put the main execution logic of the program into the if __name__ == '__main__':
• you don't need that pass in the get_list_of_urls() function
• depth = depth + 1 could be shortened to depth += 1

Performance

• since you are requesting the pages from the same host multiple times, consider switching to requests making use of a single session instance which allows to re-use an underlying TCP connection making subsequent requests to the same host faster
• importing SoupStrainer was actually a good idea. You can use it and scope the parsing to only the desired parts of the HTML
• since you are using regular expressions checks here and there, consider pre-compiling them and using the compiled patterns for searching and matching

Some of the general things I would work on:

• split the code into separate modules logically. Currently, you have all the code mixed up in single file - you have argument parsing, database interactions, web-scraping code blocks in one place
• consistent indentation. Use 4 spaces for indentation

###Code Style

• avoid handling broad exceptions with a bare except
• remove unused imports and re-group them based on PEP8 recommendations
• make sure to properly use whitespaces in expressions in statements
• put the main execution logic of the program into the if __name__ == '__main__':
• you don't need that pass in the get_list_of_urls() function
• depth = depth + 1 could be shortened to depth += 1

###Performance

• since you are requesting the pages from the same host multiple times, consider switching to requests making use of a single session instance which allows to re-use an underlying TCP connection making subsequent requests to the same host faster

Some of the general things I would work on:

• split the code into separate modules logically. Currently, you have all the code mixed up in single file - you have argument parsing, database interactions, web-scraping code blocks in one place
• consistent indentation. Use 4 spaces for indentation
• variable naming. Use descriptive variable names. Variable names like l, i or j are not meaningful and raise questions when reading the code

###Code Style

• avoid handling broad exceptions with a bare except
• remove unused imports and re-group them based on PEP8 recommendations
• make sure to properly use whitespaces in expressions in statements
• put the main execution logic of the program into the if __name__ == '__main__':
• you don't need that pass in the get_list_of_urls() function
• depth = depth + 1 could be shortened to depth += 1

###Performance

• since you are requesting the pages from the same host multiple times, consider switching to requests making use of a single session instance which allows to re-use an underlying TCP connection making subsequent requests to the same host faster
• importing SoupStrainer was actually a good idea. You can use it and scope the parsing to only the desired parts of the HTML
• since you are using regular expressions checks here and there, consider pre-compiling them and using the compiled patterns for searching and matching