Why is this field calculator expression evaluated incorrectly when using Python syntax?

Question

I have a polygon featureclass containing an integer field 'gridcode'. I've added 2 new float fields VB_calc and Python_calc, both of which should be equal to gridcode / 1,000,000.

If I calculate the field using the Field Calculator with the VB Script [fieldname] syntax, the result is as expected.

enter image description here

If I use the Python !fieldname! syntax, the results are all zero.

enter image description here

enter image description here

What am I doing wrong here? I entered those expressions by double-clicking the gridcode fieldname in the calculator. (My goal is to use this within a stand-alone Python script but I'm debugging it here since the script isn't working either.) Both fields are definitely of type FLOAT.

• Michael Stimson: Hi Stephen, try float(!GRIDCODE!)/100000.0 - I've had issues with the python calculator and integer division and found that floating the integer types help.
• Stephen Lead: Thanks, I just reached the same conclusion while debugging in Python. inValue / 1000000 => 0 while float(invalue) / 1000000 => 0.03563. Reckon this is a bug, or a known issue?
• Stephen Lead: In Python however, integer division truncated the result to an integer - I guess it's a feature of python ;) PS put this as an answer and I'll accept it
• PolyGeo: stackoverflow.com/questions/183853/… has an answer that says: "It helps to clarify for the Python 2.x line, / is neither floor division nor true division ... / is floor division when both args are int, but is true division when either or both of the args are float." My understanding is that Python 3.x works the way you originally expected.

Answer 1

As you've discovered python truncates integers that are divided - this may include floating point fields with integers stored in them, sometimes 'duck typing' isn't your friend!

To force the issue and produce a floating point answer from a division float your number first:

float(!GRIDCODE!)/100000.0 

It is best to do this always when dividing otherwise if your enumerator (value in the field) is an integer you could get a truncated answer thanks to pythons' sometimes handy duck typing of variables.

For example if most of your values are like 999.999999 but one is actually 1000 and you divide by 10000 you will get around 0.99999999999 as the answer for most and 0 for the one that has an integer value...