Why does ArcGIS Field Calculator expression unexpectedly return a binary value (0 or 1)?

Question

I am running simple field calculator expressions and experiencing some of the silliest results I have ever seen.

I am working on census data (living as a shape file) and trying to perform a simple [fieldName]/[fieldName] calculation on a newly created field.

ArcGIS happens to enjoy playing around with me... and returns values 0 and 1 in no particular order/reason that I can really justify.

Both input fields from the field calculator expression are of double data types. I have created a new field of double data type as well. However, the problem persists with other data types, including long/short integer.

Is this a bug? Has anyone else experienced this? Shortest route to create + update a new field other than moving between SPSS/Excel and ArcGIS all day long?

Answer 1

If one value or another can be rounded to an integer, the field calculator will format it as an integer and the operation (multiplication, division) will revert to integer arithmetic.

>>> 1/2
0
>>> 1/2.
0.5
>>> 1./2
0.5
>>> 1./2.
0.5

The trick here is to wrap them as floating points in the expression, do float([fieldName])/float([fieldName]).

• user681: They should be float types already since it was indicated that the fields were double in at least some cases.
• Jason Scheirer: But ensuring with an explicit float call should hopefully fix it, or at least act as a diagnostic to find the real underlying issue.
• Michael Markieta: This raises a geoprocessing error.. type mismatch 'float'

Answer 2

By not specifying the Precision and Scale of the new field, the calculation returned whole integers of 0 and 1. Creating a new field of double data type and Precision = 11 and Scale = 11 worked just fine.

However, if I were to create a copy of both [fieldName] in a new float data type field and perform the calculation, this solved the problem. Note that I didn't have to specify the precision or scale when doing so.

• user681: When creating a double field, you should specify a precision (aka field width) and scale (aka number of decimal places). I don't know what the defaults that arc uses but obviously they aren't sufficient.
• Michael Markieta: 0 and 0 are the defaults, go figure!