Place potatoes in a large pot and cover with cold water. Bring to a boil, cover and cook until tender, about 20 minutes. Drain and cool. Cut potatoes into quarters and place in a large bowl. Add bacon bits, celery, and green onions. Stir the mustard and olive oil together in a small bowl. Pour over the potatoes and toss until coated. Season with salt and pepper. Serve at room temperature.