We study a memory-based Boolean game (MBBG) taking place on a regular ring, wherein each agent acts according to its local optimal states of the last M time steps recorded in memory, and the agents in the minority are rewarded. One free parameter p between 0 and 1 is introduced to denote the strength of the agent willing to make a decision according to its memory. It is found that giving proper willing strength p, the MBBG system can spontaneously evolve to a state of performance better than the random game; while for larger p, the herd behaviour emerges to reduce the system profit. By analysing the dependence of dynamics of the system on the memory capacity M, we find that a higher memory capacity favours the emergence of the better performance state, and effectively restrains the herd behaviour, thus increases the system profit. Considering the high cost of long-time memory, the enhancement of memory capacity for restraining the herd behaviour is also discussed,and M = 5 is suggested to be a good choice.