For a great rustic stay in caves visit Cuevas de Rolando on the edge of Guadix town. A lovely little complex of eight caves for rent or bed and breakfast in a cave.
With a heated outdoor pool, plenty of parking, great views and a friendly welcome Cuevas de Rolando make a great base to explore Guadix and the Altiplano area of Granada province.
I stayed here on 05/09/2011 and thoroughly enjoyed the stay and around. Guadix had far more on offer than caves, though the cave districts were fascinating and the Casa-cueva museum in the town absolutely brilliant.
A very pleasant and quiet stay out of high season.
For more on Cuevas de Rolando see their website, unfortunately it´s only in Spanish, if you have any difficulty, leave me a comment and I´ll translate or help if I can.