It's easiest to sex your octopus when it's sitting on the glass and you can see its arms, suckers and mouth. Mature males usually have enlarged suckers (noticably bigger than ones around it, especially compared to other arms, usually near the edge of the web). The arms they're on depends on the species, but it's usually the side arms (2 and 3). The hectocotylus can be hard to identify, but I've attached some sketches at least to help you look for it. For almost all octopuses, the hectocotylus will be the third right arm of males (it's on the left in a rare few). There's a small spoon-like tip that the males use to pass the spermatophores (sperm packages) to the female when they mate. Its size varies per species, but is likely to be anywhere from 1-2mm (in aculeatus) to about a centimeter for aquarium octos. Along the back edge (funnel-side) of the arm, look for a thin pale line. This is actually part of an unpigmented "tube" through which the spermatophores are passed down the hectocotylus. It can be very thin, in species where the flap is not very muscular, to pretty obvious when the flap is fleshy.
Males also tend to protect the hectocotylus by curling the tip back when they move around. This arm can also be considerably smaller than the third arm on the left side. Ok, happy sexing!