Existing clothes retrieval methods mostly adopt binary supervision in metric learning. For each iteration, only the clothes belonging to the same instance are positive samples, and all other clothes are“indistinguishable”negative samples, which causes the following problem. The relevance between the query and candidates is only treated as relevant or irrelevant, which makes the model difficult to learn the continu-ous semantic similarities between clothes. Clothes that do not belong to the same instance are completely considered irrelevant and are uni-formly pushed away from the query by an equal margin in the embedding space, which is not consistent with the ideal retrieval results. Moti-vated by this, we propose a novel method called semantic-based clothes retrieval (SCR). In SCR, we measure the semantic similarities be-tween clothes and design a new adaptive loss based on these similarities. The margin in the proposed adaptive loss can vary with different se-mantic similarities between the anchor and negative samples. In this way, more coherent embedding space can be learned, where candidates with higher semantic similarities are mapped closer to the query than those with lower ones. We use Recall@K and normalized Discounted Cu-mulative Gain (nDCG) as evaluation metrics to conduct experiments on the DeepFashion dataset and have achieved better performance.