Smallest envirionment is a good and common question and we are thinking about a way to consolidate some of the beginner FAQ's to provide consistent and extended answers for the new year.
The smallest octopus we commonly keep is the Caribbean O. mercatoris. It is a relatively inactive species, noctural and has short arms (a consideration when sizing a tank). IME, a 20 gallon tank with a HOB overflow skimmer/filter, edging around the top with a removable lid placed over the remaining opening and lowering the water roughly 1.5 to 2" with regular weekly water changes will work well for this animal. There lifespans are generally somewhat shorter than the larger animals. I have kept two males a full 13 months (tank born so I knew the age) but the females seem to live only about 8 (8-12 months is the generally listed lifespan). They are typically not very interactive and many keepers are disappointed when they keep them because they are seldom seen. A tank set up this way will accomodate two (but you cannot go smaller for one) if they are sibblings or found together in the wild (there may be other combinations but this has proven to work, if they are unknown to each other, then they should at least be the same size but the results have been mixed). Anything larger should be housed in a 50-65 gallon tank.
I do not keep cuttlefish but will pass on what has been generally recommended. The only cuttles we commonly see are S. bandensis. These are a small cuttle that grows between 4" and 6" adult size. They have done best when raised from egg (expensive live foods required the first month or so) and will need a smaller environment until they are eating well on frozen foods (generally at about 1.5" in length). Often, a breeder net is used until they can be freed to the larger environment. One can be kept in a 30 gallon tank but preferences have been to keep multiples and a minimum 55 gallon tank is recomended when housing 2-3 animals.
Keep in mind that both cephs live short lives and when you set up a tank you are designing it for different occupants each year. A 55+ (with sump if you heve the room) is a better choice for an on-going ceph tank. If you design it for octopuses, you can experiment with keeping both octos and cuttles over the life of the tank.
Cuttles can be tank bred and occasionly we have success with a couple of the large egg species of octopuses (O. mercatoris is large egged). Mercs will tank breed and we have several successes with multiple generations but larger octopus is problematic in that keeping them together often results in the loss of one of the octopuses. I have tried breeding a pair of tank raised (from a fertile WC mother) sibbling O. brieareus, giving supervised visitation (Roy breeds a number of different species in the Berkeley lab in a temporary pairing environment). The fertilization was successful but the hatchlings did not survive. Bendensis cuttles do much better together. There are still problems in smaller environments with fatal fighting but much less than with octopuses. (Note this requires a larger tank than the single occupant size mentioned) and will breed and lay eggs throughout their adult lives. The octopuses we commonly keep lay only one clutch of eggs and the female dies about the time of hatching (she will typically lay eggs, fertile or not, and die when the eggs would have hatched even if she has not mated).
I am not familiar wiht the red PAR38 lighting but note that the corals that can be kept with octopuses are limited to very low stinging varieties (like leathers, gorgonian, a very few polyps, sponges and mushrooms). Daytime lighting to support this limited set is fine but provide dim red light for night viewing. Cuttlefish are much more tollerant of corals (I'll let the keepers be more specific as to the kinds, the ones mentioned for octos are fine but I don't have the experience to list others) and have done well even under metal halide lighting as long as darker shaded areas created by your aquascaping are provided.