Python list.append output values differ from list.extend

Question

Saw a question on another site about a piece of Python code that was driving someone nuts. It was a fairly small, straightforward-looking piece of code, so I looked at it, figured out what it was trying to do, then ran it on my local system, and discovered why it was driving the original questioner nuts. Hoping that someone here can help me understand what's going on.

The code seems to be a straightforward "ask the user for three values (x,y,z) and a sum (n); iterate all values to find tuples that sum to n, and add those tuples to a list." solution. But what it outputs is, instead of all tuples that sum to n, a list of tuples the count of which is equal to the count of tuples that sum to n, but the contents of which are all "[x,y,z]". Trying to wrap my head around this, I changed the append call to an extend call (knowing that this would un-list the added tuples), to see if the behavior changed at all. I expected to get the same output, just as "x,y,z,x,y,z..." repeatedly, instead of "[x,y,z],[x,y,z]" repeatedly, because as I read and understand the Python documentation, that's the difference between append and extend on lists. What I got instead when I used extend was the correct values of the tuples that summed to n, just broken out of their tuple form by extend.

Here's the problem code:

my = []
x = 3
y = 5
z = 7
n = 11
part = [0,0,0]
for i in range(x+1):
 part[0] = i
 for j in range(y+1):
 part[1] = j
 for k in range(z+1):
 part[2] = k
 if sum(part) == n:
 my.append(part)
print(my)

and the output:

[[3, 5, 7], [3, 5, 7], [3, 5, 7], [3, 5, 7], [3, 5, 7], [3, 5, 7], [3, 5, 7], [3, 5, 7], [3, 5, 7], [3, 5, 7], [3, 5, 7], [3, 5, 7], [3, 5, 7], [3, 5, 7]]

And here's the extend output:

[0, 4, 7, 0, 5, 6, 1, 3, 7, 1, 4, 6, 1, 5, 5, 2, 2, 7, 2, 3, 6, 2, 4, 5, 2, 5, 4, 3, 1, 7, 3, 2, 6, 3, 3, 5, 3, 4, 4, 3, 5, 3]

And the extend code:

my = []
x = 3
y = 5
z = 7
n = 11
part = [0,0,0]
for i in range(x+1):
 part[0] = i
 for j in range(y+1):
 part[1] = j
 for k in range(z+1):
 part[2] = k
 if sum(part) == n:
 my.extend(part)
print(my)

Any light that could be shed on this would be greatly appreciated. I've dug around for a while on Google and several Q&A sites, and the only things that I found regarding Python append/extend deltas are things that don't seem to have any relevance to this issue.

{edit: environment detail}

Also, ran this in both Python 2.7.10 and Python 3.4.3 (cygwin, under Windows 10 home) with the same results.

Answer 1

extend adds items from the parameter list to the list object making the call. More like, dump objects from one list to another without emptying the former.

append on the other hand, just appends; nothing more. Therefore, appending a list object to another list with an existing reference to the appended list could do some damage - as in this case. After the list has been appended, part still holds a reference to the list (since you're modifying in place), so you're essentially modifying and (re-)appending the same list object every time.

You can prevent this by either building a new list at the start of each parent iteration of the append case.

Or by simply appending a copy of the part list:

my.append(part[:]) 
my.append(list(part))
my.append(part.copy()) # Python 3 only

This will append a list that has no other existing reference outside its new parent list.

• SteveC: Thanks; I didn't grok that nuance from the documentation. Sanity restored! :-)
• SteveC: Trying to use part.copy resulted in "AttributeError: 'list' object has no attribute 'copy'"; did a little more digging, and came up with "my.append(list(part))", which solves the problem as well, by getting the list of the current part pointer rather than appending the part pointer itself.

Answer 2

There are a couple of things going on - the difference between append and extend, and the mutability of a list.

Consider a simpler case:

In [320]: part=[0,0,0]
In [321]: alist=[]
In [322]: alist.append(part)
In [323]: alist
Out[323]: [[0, 0, 0]] 

The append actually put a pointer to part in the list.

In [324]: alist.extend(part)
In [325]: alist
Out[325]: [[0, 0, 0], 0, 0, 0]

extend put the elements of part in the list, not part itself.

If we change an element in part, we can see the consequences of this difference:

In [326]: part[1]=1
In [327]: alist
Out[327]: [[0, 1, 0], 0, 0, 0]

The append part also changed, but the extended part did not.

That's why your append case consists of sublists, and the sublists all have the final value of part - because they all are part.

The extend puts the current values of part in the list. Not only aren't they sublists, but they don't change as part changes.

Here's a variation on that list pointer issue:

In [333]: alist = [part]*3
In [334]: alist
Out[334]: [[0, 1, 0], [0, 1, 0], [0, 1, 0]]
In [335]: alist[0][0]=2
In [336]: part
Out[336]: [2, 1, 0]
In [337]: alist
Out[337]: [[2, 1, 0], [2, 1, 0], [2, 1, 0]]

alist contains 3 pointers to part (not 3 copies). Change one of those sublists, and we change them all, including part.

• SteveC: Thank you for the detailed explanation; it makes significantly more sense now.