Delays to an historic restoration project on the Powys border are a "cause for concern", volunteers say.