Have you made a thermal model of the system? Doing so will go a long ways in helping understand and address your issues. Monitor your hot-side temperatures also; the temperature you see at the cold side is directly related to what’s present on the hot side.
It’s pretty common for folks to underestimate the amount of hot-side heat sinking required to achieve any significant refrigeration effect with a TEC. They’re really much more effective as heaters, because every electrical watt one puts in comes out the hot side as heat. Not so from a cooling perspective; one might have to put two or three electrical watts in, in order to suck just one thermal watt off the cold side.
The best cooling value for one’s electrical money when using TECs usually comes at around 30-40% or rated maximum electrical input. Beyond that, there’s a pretty sharp schedule of diminishing returns; the difference between operating at 80% and 100% of rated electrical input might be only a 3-4% increase in potential cooling performance.
Long story short, the standard suggestions for achieving lower cold-side temperatures with a thermoelectric cooler are backing off from maximum rated electrical inputs, using a more capable hot side heat sink, and a bigger fan to cool it.