In this paper, we propose Text-based Open Molecule Generation Benchmark(TOMG-Bench), the first benchmark to evaluate the open-domain moleculegeneration capability of LLMs. TOMG-Bench encompasses a dataset of three majortasks: molecule editing (MolEdit), molecule optimization (MolOpt), andcustomized molecule generation (MolCustom). Each task further contains threesubtasks, with each subtask comprising 5,000 test samples. Given the inherentcomplexity of open molecule generation, we have also developed an automatedevaluation system that helps measure both the quality and the accuracy of thegenerated molecules. Our comprehensive benchmarking of 25 LLMs reveals thecurrent limitations and potential areas for improvement in text-guided moleculediscovery. Furthermore, with the assistance of OpenMolIns, a specializedinstruction tuning dataset proposed for solving challenges raised byTOMG-Bench, Llama3.1-8B could outperform all the open-source general LLMs, evensurpassing GPT-3.5-turbo by 46.5% on TOMG-Bench. Our codes and datasets areavailable through https://github.com/phenixace/TOMG-Bench.