A legspinner from Kumble trapped Andrew Hall leg-before-wicket with a googly for a duck.
Kumble then bowled South Africa's Andre Nel for another duck with another googly three balls later in the day's 73rd and final over.
Strong off-sider Ashwell Prince was not out 98 and Morne Morkel was yet to score at the stumps at Kingsmead's Sahara Stadium.
South Africa win toss
South Africa attempted to rally, after losing the first test by 121 runs, with captain Graeme Smith winning the toss and electing to bat.
India's opening bowlers dominated the morning session, taking three wickets to leave South Africa on 67-3 at lunch.
With Jacques Kallis out with a strained lower back, it was going to be important for South Africa's top order to score runs, but Smith had scored just 5 before he mis-timed a pull in the third over off Zaheer Khan and Sachin Tendulkar took a running catch from first slip.
|South Africa's Prince sweeps India's Kumble|
Hashim Amla was then LBW by Khan for one, before AB de Villiers, promoted to opener ahead of Herschelle Gibbs, was caught by Tendulkar for nine while attempting a drive off Sreesanth.
Prince, dropped on 41 by Tendulkar off Vikram Singh's bowling, had hit 13 fours in 181 balls.
He put on a 94-run, fourth-wicket stand with Gibbs and 100 with Mark Boucher for the fifth wicket.
Gibbs had hit 13 boundaries in 88 balls, but was caught behind by Mahendra Singh Dhoni off Sree Sreesanth for 63 as he edged an attempted hook shot.
Boucher was bowled by Sreesanth for 53 and Shaun Pollock was well caught by Virender Sehwag at backward point off a ball from Singh.