Tracebacks in python 3.5

Question

To handle the errors in Python 3.5 gracefully, I am using the following code:

 except:
 exc_type, exc_value, exc_traceback = sys.exc_info()
 # print("Unable to write feed_item in database")
 # print(feeditem_insert_data)
 error_insert_data = (
 prog_name, 'Unable to populate item for (feed id,ref_feed_id) ('
 + str(feed_id) +',' + str(ref_feed_id) + ') in feed_item table'
 , str(exc_type), str(exc_value), repr(traceback.format_tb(exc_traceback)), datetime.datetime.now()
 )
 cur_error_log.execute(error_insert_stmt,error_insert_data)
 continue

After adding this code at many places in my program, I am observing weird behavior as in program running long behaving on same data differently than program running short. I cant conclusively say this behavior showed up after adding the code above.

But I remember reading somewhere that if I use traceback than I have to deallocate memory manually.

Please advice if the module I have above is technically correct or something needs to be done to avoid issues.

• gil: If continue is part of what you are adding, it's either superfluous or will change the loop behavior. But otherwise it's impossible to diagnose what's wrong (if anything is) without the rest of the code.
• Peter: Sorry I dont understand, please help me understand. Continue is to ignore this record and jump to next one for processing - how can it be harmful or related to diagnosis.
• gil: If you have some more code after the except block, then continue in the except block will change how you loop when an exception occurs: without it, the code below will be executed after the exception is handled; with it, the code below will be skipped. If you don't have any more code after the except block, then continue makes no difference and is superfluous.
• Peter: Yeah.. I do have code - that's why continue. Thanks for reply though.

Answer 1

The main change in Python 3 regarding exceptions is that it's no longer necessary to use sys.exc_info in order to get the traceback. It's now available as the __traceback__ attribute on the exception object (this is possible since all exceptions in Python 3 must inherit from the base Exception class, while older versions of Python 2 allowed anything to be treated as an exception).

So, you could replace the first two lines of your except code with except Exception as e: and then use type(e), e, and e.__traceback__ in the error reporting code, rather than the values you were getting from sys.exc_info.

Having the traceback as an attribute on the exception can have some adverse consequences though on memory usage. Specifically, it creates a reference cycle between the current frame of execution and the exception. The frame references the exception (if you've bound it to a local variable), the exception references the traceback through its attribute and the traceback references the frame. The cycle means that the memory can't be reclaimed immediately by the reference counting system when it goes out of scope. Instead it has to rely upon the cyclical garbage collector, which might be slow to reclaim the memory, or even never get around to it (you can turn it off completely).

To reduce the impact of this issue, Python will automatically delete the exception variable created by a except ExceptionType as e statement at the end of its block. If you create references to the exception under some other name, you may need to clean them up yourself to avoid the memory sticking around longer than you need it.

So, if you really are having exception related memory issues, try adding del exc_value, exc_traceback at the end of the try block, just before continue. Or alternatively, switch to the except Exception as e syntax I described above, and e (or whatever name you use) will be removed from the local namespace for you automatically.

• Peter: Thank you so much for the detailed reply. I think del changed the behavior. I am going to try other suggestion out as well.
• WloHu: Are there any references which explain this implementation detail of removing as exc name from except? I know it works that way because I checked it in Python.
• Blckknght: @JCode: The behavior is documented in the language reference.