Who makes the best reference? The best reference is someone who managed your work and will speak glowingly of you. A good reference-checker is looking to see how your past managers talk about your work, and whether they're enthusiastic about you or not. A reference who is unreservedly enthusiastic about you counts for a great deal. On the other hand, a reference who sounds hesitant, uncomfortable, ambivalent, or simply not enthusiastic can raise red flags. So you want references who will champion you and your work.
Do references have to be managers? Ideally, yes. People who managed you and therefore were charged with evaluating your work are able to speak to more of what reference-checkers want to know. Peers can talk about you as a co-worker, but managers can talk about you as an employee. Plus, if you mainly put peers or other non-managers on your reference list, employers will wonder why you don't want your past managers contacted.
What if you don't want your current manager to be contacted? Then you're just like most people. It's common to ask that your current employer not be contacted, because most people don't want to tip off their employer to their job search. In fact, doing so can even jeopardize your current job. So it's normal and reasonable to explain that your current employer doesn't know about your search and ask that no one there be contacted. If a prospective employer is insistent, you can always offer to allow them to contact your current manager once you have an offer (which can be contingent on a good reference).
Do you need to alert references that you're listing them? It's smart to alert your references that they might be contacted, because you want to verify that they're still willing to provide you with a strong reference. You also want to make sure you still have their current contact information (you risk looking disorganized if you provide a prospective employer with out-of-date information), and that your reference will be available and not out-of-town or otherwise unreachable. Plus, alerting them is a good opportunity to prep them on any key points that you want them to emphasize with the reference-checker.
When is the right time to offer references? The right time to offer references is when you're asked for them. Don't offer them before that—such as including them on your resume—because you want to know when your references are likely to be contacted, so that you can give them a heads-up. You don't want to alert them every time you apply for a job or get called for an interview, only when you're a serious finalist and the employer is at the reference-checking stage of the process.
Can you be sure that employers will only call the people on your list? No. It's important to realize that employers aren't limited to just the references you provide them with. They can call anyone at all to ask about you, so if you noticeably omit recent managers from your list, they might call them anyway or ask you to put them in touch. And a lot of reference-checking happens behind the scenes when an employer spots a mutual connection and calls that person to ask their opinion of you. The only reference who is typically considered off-limits is your current employer.
But many employers won't bother going outside your list, so it's important to choose your list with care.
Alison Green writes the popular Ask a Manager blog, where she dispenses advice on career, job search, and management issues. She's also the co-author of Managing to Change the World: The Nonprofit Manager's Guide to Getting Results, and former chief of staff of a successful nonprofit organization, where she oversaw day-to-day staff management, hiring, firing, and employee development.