mkvtomp4 is an open source and command line based tool which uses mpeg4ip, mkvtoolnix and ffmpeg to convert troublesome mkv files to mp4.
The conversion does not re-encode the video and only re-encodes the audio if it doesn't use AAC codec (one can override this behaviour using --audio-codec). The resulting mp4 files will be playable on the Sony PS3.
Installation: To install mkvtomp4 on your Mac you will have to open a Terminal window, browse to mkvtomp4's folder and run the following command from inside that directory (you will need administrator privileges to be able to run it):
sudo python setup.py.in install