Transformers Need to Know