Dining across the divide: ‘The young have so much idealism – but for him everything was determined by getting the Conservatives elected’