How do you check for a custom banner?

To check for an item in the players hand:

/testfor @p[r=10] {SelectedItem:{id:"minecraft:diamond_sword"}}

Or if you want to check the players entire inventory:

/testfor @p[r=10] {Inventory:[{id:"minecraft:diamond_sword"}]}

To specifically target a type of banner you need to know the exact pattern of the banner, then you can use this structure to find it:

/testfor @p[r=10] {Inventory:[{
    id:"minecraft:banner",
    tag:{BlockEntityTag:
        {Base:15,
        Patterns:[
            {Pattern:mc,Color:4},
            {Pattern:flo,Color:15},
            {Pattern:tt,Color:4}
        ]}
    }}
]}

To break this command down, once you've selected the banner item, you need to know the banners base colour (in this example 15) then you match each pattern on the banner inside the pattern array.

Each pattern looks like this {Pattern:mc,Color:4} including the the pattern and the colour. Stack these commands up so they exactly describe your banner, and it should correctly check if the player tested is carrying it.

Command as you would use it (edited to match your banner of course) would look like this:

/testfor @p[r=10] {Inventory:[{id:"minecraft:banner",tag:{BlockEntityTag:{Base:15,Patterns:[{Pattern:mc,Color:4},{Pattern:flo,Color:15},{Pattern:tt,Color:4}]}}}]}

Some good resources:

http://www.minecraftforum.net/forums/minecraft-discussion/recent-updates-and-snapshots/2179208-testfor-custom-banner-on-a-players-head