Cred to Eaton! I've never heard the alternatives-to-core-blocks functions of Panels vs. Context described so clear and compact as just after 47 minutes into the podcast.

I am most happy to quote:

[In Panels] you make a panel that lives in a place and it can get information about the details of how it is being used at a particular moment. Like, “I'm a panel that displays nodes – what node is being displayed, what are the properties of that node – I can switch and make different decisions based on that”.
Context on the other hand is not necessarily tied to any particular URL that you are creating on the website. Rather it it's about setting up rules saying that “at URLs that match this particular pattern, I'm going to carve out a particular space on my website and call that the new section” […]. Context looks at the environment and carves out that information about what's currently going on, and then things can look at that and respond to that by changing the theme or changing what block are visible […].
So, Contexts and Spaces sort of are layers that goes on top of what's going on the site and exposes more information and lets you set up rules that define areas and zones on your website, whereas Panels goes a little deeper and says “OK, I'm going to be the traffic cop for this, and along the way I'm also going to extract information about what cars I'm letting through and rerouting […], and allow people to make decisions based on that”.

If it wasn't so darn long, I'd print it on a t-shirt and wear at DrupalCon. :-)