For parts, I'd recommend RadioShack. Pretty much everything you need from solder, to bulbs, wire, resistors, etc.
First thing you need to know about LEDs is they have fairly low electrical tolerances; apply too much power and they will burn out fast. Resistors are going to be your most important component.
This is a great resource for making sure you have the right size resistor for your LEDs. Plug in the numbers and it tells you the size you need; if ever in doubt about a resistor, go up a size to be safe.
When it comes to raw LEDs, the long lead is typically power and the short is ground.

The way power flows, you want the "hot" side of your power source (typically a red wire leading from the battery hookup) going to the long lead and a wire going back to ground on the battery (usually a black wire).

You can add a switch to either end of the circuit, power side
or ground side; the light will only function if there is a direct path for power to ground, and the switch will break that link regardless of where you place it.