Unlike general in-game voice communication, a chat is a closed conversation between you and one other person. Your chat partner can be playing the same game you are, but can also be playing a different game, or watching a DVD movie, or listening to music. Both of you could, in fact, just be sitting around wondering what to do next.
Xbox 360 private chat lets you send and accept invitations to chat right from the Xbox Dashboard. Here is a step-by-step guide: