Federer went down 7-5 7-5 to Thiem, who has failed to qualify from the group stages in his three previous appearances at the end of season showpiece.
But the fifth seed went into the match with a 4-2 career win record over the veteran Federer, who was nonetheless the favorite after his recent tournament victory in Basel.
The Swiss 20-time grand slam winner made the worst possible start as he was broken in the first game of the match, but broke back to level in a closely fought opener.
Thiem then broke again in the 11th game before serving out to take a one set lead.
Federer failed to take advantage of an early chance to break in the second set before it was the 11th game again which proved decisive as he was easily broken.
Third seed Federer forced two break points as he tried desperately to stay in the match, but Thiem kept his nerve to win on his second match point.
"It's amazing to play against probably the best player of all time, but luckily I have a pretty good record against him," said Thiem.
Federer, who last won the tournament back in 2011, now faces an uphill task to move out of the Bjorn Borg group, which also includes Novak Djokovic and Matteo Berrettini. He has not failed to reach the knockout stages of the tournament since 2008.
Djokovic bids for records
Six is the magic number for Djokovic at the ATP Finals this week and the Serbian wasted little time in securing his opening victory of the round-robin section earlier Sunday.
Djokovic is aiming to end the year as No.1 for a sixth time, equaling the record of American great Pete Sampras, and for his sixth title at the season-ending finale, matching Federer.
He made light work of Berrettini, winning 6-2 6-1 in just over an hour at the O2 Arena in their group opener.
Berretinni, the first Italian man to reach the year-end championships since Corrado Barazzutti in 1978, came out fighting and showed early signs of his quality to hold for one game all.
But he was undermined by 17 unforced errors in the first set as nerves played their part, allowing Djokovic to break twice and seal the set in 28 minutes.
Eighth seed Berrettini produced some of his best play at the start of the second, but it could not prevent him being broken twice as he trailed by a set and 4-0.
Djokovic dropped his own serve to give Berrettini a glimmer of hope, but a third break of his 23-year-old opponent settled any nerves and he served out for victory in 62 minutes.
"It was not easy for him at his first Finals match, I knew he would be nervous at the beginning and I used my experience to perform well," said Djokovic as he reflected on a comfortable victory.
Nadal faces tough opener
To have a chance of overtaking Rafael Nadal at the top of the rankings Djokovic must at least reach the final, but he has made an imposing start.
Nadal, who had to pull out of the Paris Masters, won last week by Djokovic, with an injury that affects his service, plays defending champion Alexander Zverev in their Andre Agassi Group opener Monday.
Germany's Zverev stunned Djokovic in last year's final to lift the title, but rather than act as a springboard for grand slam success it has been the old guard of Nadal and Djokovic who have shared those majors in 2019.
Stefanos Tsitsipas of Greece and Russian Daniil Medvedev are the other qualifiers for the eight-man tournament.