We consider rotational motion of an interacting atomic Bose–Einstein condensate (BEC) with both two- and three-body interactions in a quadratic-plus-quartic and harmonic-plus-Gaussian trap. By using the variational method, the influence of the three-body interaction and the anharmonicity of the trap on the lowest energy surface mode excitation and the spontaneous shape deformation (responsible for the vortex formation) in a rotating BEC is discussed in detail. It is found that the repulsive three-body interaction helps the formation of the vortex and reduces the lowest energy surface mode frequency and the critical rotational frequency of the system. Moreover, the critical rotational frequency for the vortex formation in the harmonic-plus-Gaussian potential is lower than that in the quadratic-plus-quartic potential.
The behaviors of a rapidly rotating Bose-Einstein condensate under extreme elongation in a two-dimensional anisotropic harmonic plus quartic trap are investigated. Due to the quartic trap, the system remains stable at high rotating velocity, $\ensuremath{\Omega}\ensuremath{\geqslant}{\ensuremath{\omega}}_{\ensuremath{\perp}}$ (${\ensuremath{\omega}}_{\ensuremath{\perp}}$ is the radial harmonic trap frequency), and vortex lattices form even in the absence of the repulsive
s-wave interaction (
g). When
g is present, the interplay between
g and the quartic trap potential can lead to rich vortex lattice transition states as a function of $\ensuremath{\Omega}$, to which vortex lattices vanish eventually at some higher $\ensuremath{\Omega}$.