"We are going down to a number less than 5,000 before the end of November," Esper said in an interview with Fox News.
Esper said the Pentagon would still need to brief members of Congress on the plan, and would also need to insure the "United States is not threatened by terrorists coming out of Afghanistan."
In an interview with Axios last week, President Donald Trump said he would like to have "probably anywhere from 4,000 to 5,000" troops in Afghanistan by the time of the election on November 3.
Esper's confirmation of a major drawdown comes amid ongoing deadly Taliban attacks against the Afghan government, despite the insurgent group signing an agreement with the US in February.
Despite the lack of progress, on Thursday a Pentagon spokesman insisted while the President's guidance was clear, any future drawdowns would be "conditions based."