Guess the backs of US bills - a quiz

Mental Floss has a quiz in which you must guess the denomination of the U.S. bill from the back without any additional information. I know most of us know the fronts, but how often do you examine the back of a bill, and then memorize it in case there is a quiz.

I got them all!

Bill Backs
Score: 100% (5 out of 5)

Go try the quiz and then come back here and I will tell you how I did it and why I think the quiz is easier than it appears at first glance.

I will put the spoilers below in a similar color to the background so as not to tempt you before you take the quiz, highlight to read.

You see, I tried to think of what would be an appropriate back to a bill, given the president on the front. The key is to remember who is on the bill, and recognize the building.

Easy to remember is Lincoln on the $5, Franklin on the $100 (it's all about the Benjamins, baby), and Hamilton on the $10 (You can call us Aaron Burr / From the way we're droppin' HAM-IL-TONS). Thus Lincoln and the $5 should have the Lincoln Memorial, Franklin and the $100 should have Philadelphia's Independence Hall, and Hamilton should have the Treasury building. Which one is that? It's the one you don't recognize. Then I guessed the other two. The White House should be for a president, but everyone left is a president. Lastly is the United States Capitol building. Thus the $20 and the $50 are the toughest to guess. Looks like I guess right.

It is like any other puzzle, if you don't remember it straight on, what are some other clues that you can use.