Manuel Lanzini's penalty midway through the second half looked to have set West Ham on the way to a memorable victory in north London.
But in a dramatic turn of events in the final stages of the game, Kane managed to score two goals in the space of two minutes to help Tottenham secure all three points.
Michail Antonio had fired West Ham into the lead in the first half, but youngster Harry Winks marked his Premier League debut with an equaliser.
Lanzini then scored from the penalty spot after Winston Reid had been brought down inside the box.
But the Hammers failed to hold on and Kane brought Tottenham level on 88 minutes before firing the winner from the penalty spot.
Bilic made three changes to the team which drew with Stoke in the last game handing first Premier League starts of the season to Darren Randolph and Diafra Sakho and Winston Reid returning to the starting line-up after a one match ban.
West Ham fans were especially thrilled to see Sakho back in action after he was forced to miss the start of the season with a back injury.
The home side thought they had opened the scoring after just six minutes.
Eric Dier played a ball over the top of the Hammers defence. Christian Eriksen reacted the quickest and fired a close-range effort past Darren Randolph, but his effort was ruled out for offside.
West Ham made them pay as they broke the deadlock on 24 minutes and it was another moment for Antonio to remember against Spurs.
Dimitri Payet's corner picked out Cheikhou Kouyate who saw his looping header strike the bar.
The rebound fell to Winston Reid who saw his shot flicked home by Antonio at the far post.
It was yet another header from Antonio who took his tally to six goals this season and now just three away from equalling the Premier League record of headed goals in one season by former Everton forward Duncan Ferguson.
Tottenham drew level on 51 minutes after Randolph could only parry a close-range shot from Vincent Janssen. The rebound fell to Winks who made no mistake striking the ball into the back of the net.
Randolph kept the Hammers in the game on 58 minutes when he made a fine save to deny Dier's close-range header.
West Ham made Tottenham pay as Lanzini added a second from the penalty spot on 66 minutes.
Janssen was penalised for pulling back Reid. Referee Mike Dean had no hesitation in pointing to the spot and Lanzini fired the spot kick into the corner of the net.
It looked as if the Hammers were heading for all three points, but Tottenham somehow managed to turn the game around right at the death.
Kane helped Tottenham draw level on 88 minutes when he turned in a cross from substitute Heung-Min Son.
And the Tottenham forward scored the winner from the penalty spot in injury time after Nordtveit had brought down Son inside the box.
The Hammers misery was compounded in injury time when Reid was sent off for two bookable offences.
Tottenham: Lloris, Walker (Trippier 89), Rose, Vertonghen, Dier; Wanyama, Dembele (Son 72), Winks, Eriksen; Kane, Janssen (Alli 67)
Subs not used: Vorm, Trippier, Onomah, Wimmer, Carter-Vickers
Bookings: Dembele, Rose
West Ham: Randolph; Reid, Cresswell, Kouyate, Obiang, Ogbonna; Lanzini, Payet (Nordtveit 83), Sakho (Zaza 60); Ayew (Fernandes 60), Antonio
Subs not used: Adrian, Feghouli, Collins, Fletcher
Bookings: Lanzini, Antonio, Nordtveit, Reid
Sent Off: Reid
Referee: Mike Dean