You're right. From wikipedia "In the 2012 James Bond film Skyfall, the island served as an inspiration for the lair of villain Raoul Silva but filming did not take place on the island itself. One section was recreated at Pinewood Studios in Great Britain and the rest via CGI"