Iceland seems to have a reputation for high prices but after the financial troubles that hit Iceland in October 2008 the krona fell dramatically making it much cheaper to visit. Unfortunately the krona has been rising again after currency controls were lifted and the tourist industry booming and now in 2017 is at the same level as just before the crash.
Out in the countryside all the vital requirements can usually be found in the village co-op or supermarket.
In Reykjavik the major outdoors equipment shops are: Útilíf, Everest, Seglagerðin Ægir and 66°North There one can find all the necessities for outdoor sports and traveling. The bicycle shops also have some outdoors equipment. These shops are usually open from 10:00 until 18:00 during weekdays.
Powdered outdoors food is quite expensive and hard to find outside Reykjavik, but you can usually buy packets with pasta-and-sauce, flavored rice, soups etc.
The interior is uninhabited so you need to bring everything you need for the journey. You will not be able to buy supplies or anything else in the interior. The campsites have safe drinking water from the tap. Tap water is safe and no reason to buy bottled water.
If you need specialized fuel for your camping stove your best bet is to stock up in Reykjavik. Butane gas is usually available in gas stations, both self-sealing cannisters and the pierced kind used in Primus stoves, but very small hiking cannisters may be harder to locate.