Functional-style: flattening of lists of dictionaries (in Python)

(March 2012): A transcript from a Skype dialog

A friend of mine is learning Python. So far, he has only worked with imperative-style languages like C and Ada, and Python is the first language he meets where the functional style of problem-solving is easily accessible - with immediate, tangible results.

He asked me today (over Skype) for help with this code:

...
def foo():
    ...
    res={}
    for elem in t:
        res[elem.keys()[0]]=elem[elem.keys()[0]]
    return res
    ...

My first reaction, was... head scratching.

"What exactly is it that you want to do?", I asked. "You seem to be iterating over a list of dictionaries, reading the first element of the keys() of each one of them... and assigning its value... to another dictionary? You know, there's no order preservation in dictionaries - if you depended on elem.keys()[0] to give you the first element you inserted..."

He replied: "No, each of the dictionaries in the list has only one [key,value] pair. I just want to "collapse" them all into one big dictionary."

"And I imagine you can't do that when creating the list of smallish one-element ones?"

"No, I can't."

"If you are sure that only one pair is ever inserted, you could have just used a list of tuples, you know."

"I know - but can you please answer my question? How would you write this code using a lambda?"

"Well before going into lambda-stuff, even in imperative style, this would be much better:"

...
def foo():
    ...
    res={}
    for elem in t:
        res.update(elem)
    return res
    ...

"I see - I didn't know about .update. And this inserts all elements of elem into res?"

"Yes - and is much cleaner than an element assignment, no?"

"Yes, indeed. But what about a lambda version?"

(Looks like my functional-style preaching has raised his interest :‑)

"OK, let's look at this. You know about reduce, yes?"

"Well..."

"It's simple: you define a function/lambda that takes two params, and returns one: the result. You then pass it to reduce, alongside the list of things it will work on... look:"

Python 2.6.6 (r266:84292, Dec 27 2010, 00:02:40)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> listOfInts = range(10)
>>> listOfInts
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> print reduce(lambda a,b: a+b, listOfInts)
45
>>>

"See? The lambda just adds the two parameters and returns the sum... reduce will therefore do (((0+1)+2)+3)... Makes sense?"

"Sure - and what kind of lambda would work for my problem, the flattening of dictionaries?"

"Well, can you think of any operator/member that is close to what we want to "reduce" with?"

"Does '+' work on dictionaries?"

"No. Anything else?"

"Can we use .update?"

"Let's see. Does .update return the "collapsed" dictionary?"

Python 2.6.6 (r266:84292, Dec 27 2010, 00:02:40)
...
>>> a={1:2}
>>> b={4:5}
>>> print a.update(b)
None
>>> a
{1: 2, 4: 5}
>>>

"...Nope, it returns "None". So we can't use it as-is - it modifies in-place the object we call it on. Any thoughts...?"

...

"Remember the semantics of "a and b" and "c or d"?"

...

"How if c is None..."

"Of course! If c is None, the expression 'c or d' will return with 'd's result!"

"Exactly! So we could write this:"

...
>>> c=[{1:2},{4:5},{8:9}]
>>> print reduce(lambda a,b: a.update(b) or a, c)
{8: 9, 1: 2, 4: 5}
...

"a.update(b) will fill 'a' with 'b's contents, return "None", so the "or a" will return 'a' - our accumulator"

"It worked!"

"Yes :‑) But there's one more thing. Now that that is finished, check the contents of c[0]."

>>> print c[0]
{8: 9, 1: 2, 4: 5}

"Remember that we used something that "mutates" its first argument - in this case, we call .update on parameter 'a'... which comes from where?"

"From the input list... so we mess up the input!"

"Remember how I said that functional style works better with immutable inputs? This is a good example of why."

"So we lose? There's no way?"

"There is, actually. You can pass your own initial value for your "accumulator" to reduce:"

...
>>> c=[{1:2},{4:5},{8:9}]
>>> print reduce(lambda a,b: a.update(b) or a, c, {})
{8: 9, 1: 2, 4: 5}
>>> print c[0]
{1: 2}
...

"Now the mutating accumulator is the '{}' we passed, not any of the list elements. Cool?"

"Amazing."

"And once you learn about reduce, you now know something important: just by looking at a piece of code, if you see reduce, you know that a list of data is "collapsed" into one result, via a function that "shrinks" two args into one - in some way. You see now how that is much clearer than mystic for-loops?"

"Yep. Brilliant. Thanks!"

Next, I must introduce him to itertools :‑)

Index

Updated: Sat Mar 8 22:58:16 2014

The comments on this website require the use of JavaScript. Perhaps your browser isn't JavaScript capable; or the script is not being run for another reason. If you're interested in reading the comments or leaving a comment behind please try again with a different browser or from a different connection.