A Gem is a rare find from mining. At first it comes from the copper vein by the garden, and only if you have the right perk: every time the vein breaks, the Miner perk gives a 20% chance of a Gem, and the Jeweler perk another 20%. Without either perk the vein never drops one. The vein never runs out, so with a perk the Gems slowly add up.
A Gem has no use until the Jewelry Table. It goes into Fine Jewelry and the Golden Talisman there, which comes with Delicate Work; the only trader who deals in Gems is the Trademaster at the port; and it cannot be studied.