Not at the races/Not in the race

If someone is not at the races, they are not performing at their best. It derives from not being in the race, which means much the same thing i.e. losing the race because one is not performing at one's best. The figurative usage of race, meaning a course of time and not a literal race between contestants, dates from the late 1500s according to the OED.