Indiana has a large number of caves that can be explored. Some rank among the larger, longer or more beautiful caves in the U.S. The cave complex at Wyandotte, a state recreation area, has guided tours from 45 minutes to 8 hours. There are also several commercial caves; Blue Springs, Squire Boone and Marengo are the main ones. But the whole south central region is riddled with caves; these offer about as genuine a "wilderness" experience as you can get. The problem is that caves are a particularly fragile environment. Just a touch on the surface of a cave can leave a film of oil that remains for years and prevents or affects the formation of cave features. Many of the people who visit caves have, intentionally or not, destroyed or vandalized them. Serious cavers tend to be very secretive about the location of "wild" caves whose owners still give permission for exploration, and I have to say that I agree with them; just as I do not list areas that the Nature Conservancy notes as "open by permission only" or "closed to visitation", I cannot in good conscience note the locations of caves that aren't specifically open to the public. The one place in the state where you can explore non-commercial caves on your own is Cave River Park north of Campbellsburg (in the Spring Mill State Park area).  If you are interested in caving, I suggest visiting Wyandotte and taking one of the spelunking tours (rather than the walking tour) to see if you can stand this claustrophobic pastime (I can only barely tolerate it myself). If you like it, check out the Central Indiana Grotto of the National Speleological Society. Additionally, there are many organized groups of cavers around the country (and the state of Indiana) that are a valuable source of information. To find cavers in your area, visit the NSS website and look for the Find A Local Caving Organization text and pickbox on the home page. Many of those groups have persons that value the opportunity to take new persons underground.

Some tips (thanks to David Everton): Each caver should three independent sources of light (for ex. main light and two backup flashlights, batteries for each, etc). Each should also have a helmet, preferably with at least one of the lights mounted to it.
The group size should be no smaller than 4 persons; in case someone is hurt, one persons stays with the injured one and the two others go for help.

Otherwise, stay out; it's dangerous and destructive.