This suggestion can work in tandem with the Artificing suggestion as linked below:
Instead of having light/heavy/force amulet, boots, gloves, and belt, treat them similar to implants or modification to the main armor type on the torso.
This will also reduce the micromanagement of crafting the right armor type for specific characters, since any non-matching armor types on any of the equipped pieces may disable perks that are dependent on armor types. (Ex. Rapid Shot/Reload only available when in light armor, Expertise only available in heavy armor)