There is no limit on the memory pool size, but there is effective limit on the number of transactions received from other nodes. There are two pools which have limits:
orphan pool - transactions with unknown inputs. The limit is 1000, but it can be adjusted by -maxorphantx runtime parameter.
inventory pool - tx hashes which we are aware of, but didn't receive yet. The limit is 50000 per node and cannot be modified by runtime parameter