I thought about it a while ago... once.
Usually it comes down to bluff and sense motive rolls, but a good DM should obviously award good planning and roleplaying with all kinds of situational modifiers.
Seems like some kind of "social combat" system might be an interesting idea here.
I haven't seen the BBC show, but if it's anything like Leverage then I kinda get what you're going for:
The GM prepares a mark for the PCs to swindle. The PCs come up with a plan for how to accomplish this. The GM throws in some unforseen factors the PCs have to deal with and assuming they are successful then events proceed according to the plan, culminating in the successful heist.
I've not run adventures with this concept in mind, so this is pretty much how I imagine it 
:
The imporant thing here is the Plan. Once laid out and agreed upon, this is the framework of the adventure. Unlike most adventures, a lot of the creative effort has to come from the players, and both the GM and players need to be mindful of this.
The Plan is what the adventure is all about, and the success of the adventure is to be measured against how completely the Plan played out. The Plan itself likely involves a few encounters in itself, but these should be automatically successful: They are after all part of the plan.
The real challenge is when the GM throws the PCs a couple of curve-balls and introduces unforeseen elements. Obviously these kind of surprises could be of the kind that force the PCs to automatically abandon the plan (e.g. revealing the PCs to the mark directly), but in keeping with the theme of the game these encounters should at worst complicate the plot.
In the end, only the final encounter in the Plan determines wether success or failure, with successful encounters reinforcing the chance of success and failed encounters counting against the PCs in some way.
I'd think the following structure would be good:
1. The GM prepares the mark, detailing his character and resources. The GM should come up with a couple of exploitable character traits for the mark. Perhaps he's a gambler, or overly cautious. Perhaps he's a collector of fine art, or completely oblivious to the actual worth of such items.
2. The PCs do their prep-work, researching the mark and figuring out the best way to "attack". This part can play out a a standard adventure where the PCs spy on the mark, interact with NPCs or even break in to spy. Just bear in mind that this part should be mostly information-gathering.
3. The PCs come up with a plan. The GM should feel free to pitch in and offer suggestions. After all, coming up with a plan isn't what is supposed to challenge the PCs, it's executing it.
The players should eventually produce a plan that detail the plan step by step as a series of encounters.
The encounters laid out in the Plan should go off without a hitch during execution. However, you may want to test the ability of the PCs to carry out their assigned tasks (e.g. having the martial artist hacking the mansion security network with a PDA) as "training room" encounters first: a great way to work in a montage of the group preparing for the heist.
The players and GM should eventually produce a document detailing the Plan to everyone's satisfaction. It is perhaps best to end the session at this point.
4. The GM should then review the plan and Prepare any surprise encounters. These encounters should give the players the oportunity to improve the chance of their plan working (or complicate things), but they should not render the Plan useless.
5. Execution and conclusion. This is the part where the PCs set the Plan in motion and let it play out. Its important that the GM and players are all in agreement that events will progress according to plan and that one doesn't simply abandon it part-way through. This is after all a team effort. The GM's surprise encounters is where the players are really put to the challenge, but even so the game should progress to the final encounter where it's all resolved.
Hm. I think I'll take my own advice and run an adventure like this and see if it really works!
-DF
The only problem I forsee is balancing the Players' ability with their Characters' abilities when coming up with the plan. Obviously, the Players' creativity is going to play a huge role in creating it, but you shouldn't completely discount the Characters during the process.
If the players come up with a crappy plan, but one of the characters is a smart/charismatic/dedicated hero with 18 in all the mental stats (impossible, I know, but bear with me for the example) and max ranks in all the appropriate skills, then the DM should aknowledge that said character would know that its a bad plan.
Or say the players want to set off the sprinkler system in the building to cause chaos. Spinkler systems don't work that way, they're all isolated and only those activated by fire directly will go off, but the players (like many people) aren't aware of this. One of the characters has ranks in arcitecture (or whatever skill would provide knowledge of this) or is an ex-firefighter, so he should know this. Thus the DM should inform the player of this information.
Essentially the DM should work with the players to ensure that the characters' abilities and knowledge play a role in the planning. Know what I mean?
I'm pretty sure resolving out the known factors in advance is the way to go.
Activating the sprinklers all at once is a good example. You can ignore the fact that these work in isolation because the initial idea is a good one: it won't hurt the story to treat them as a single system.
On the other hand it's a known factor so you can decide how to resolve this in advance. Perhaps hacking into the building's mainframe will allow you to tie the sprinkler systems together, or trigger them all at once.
So the PCs set up a simulation in their basement/warehouse/hangar, make the necessary skill checks and determine if this element of the plan is feasible or not. If the check fails then they need to come up with something else. If successful then, when it comes to playing out the plan, this works.
Ok, I'm a bit hung up on this, but as much as I like story progression, I also like to see the players work for their lunch.
-DF
Nathan Ford: Dedicated/Investigator
Sophie Devereaux: Charismatic/Negotiator (she's not a celebrity!)
Eliot Spenser: Tought/Bodyguard (or Strong/Martial Artist?)
Parker: Fast/Infiltrator
Alec Hardison: Smart/Techie
In some ways, it's the perfect d20M team because it includes a little bit of everything.
I love the idea of proceding in phases: GM provides a mark (perhaps even plays the roper, to give it verisimilitude?), then the team has to come up with a plan. The only problem is that the GM will need some time to prepare all the encounters, so you'd want either two separate games, or for the marking/planning stage to be done over email. Then, you run the game's encounters just like scenes in a show. Everyone gets their little moment to shine--a little charm, a little punching, a little hacking--and everyone drinks champaign at the end. It could be a lot of fun.
Yes! That's it exactly. In fact, Leverage is a better example than Hustle because it involves a very mixed team. Hustle is basically a bunch of Charismatics and one Techie, but Leverage has:
Nathan Ford: Dedicated/Investigator
Sophie Devereaux: Charismatic/Negotiator (she's not a celebrity!)
Eliot Spenser: Tought/Bodyguard (or Strong/Martial Artist?)
Parker: Fast/Infiltrator
Alec Hardison: Smart/Techie
In some ways, it's the perfect d20M team because it includes a little bit of everything.
I love the idea of proceding in phases: GM provides a mark (perhaps even plays the roper, to give it verisimilitude?), then the team has to come up with a plan. The only problem is that the GM will need some time to prepare all the encounters, so you'd want either two separate games, or for the marking/planning stage to be done over email. Then, you run the game's encounters just like scenes in a show. Everyone gets their little moment to shine--a little charm, a little punching, a little hacking--and everyone drinks champaign at the end. It could be a lot of fun.
I did this a while back with a D&D thief guild. It will all come down on how the players work as a team. From what I read between the lines your group is probably 30 something. That is mature enough to make it work. Go for it.