The coin, minted as gift for the Sultan of Muscat, was one of only eight made in 1834, but bears the date 1804 as the coins were no longer being made and an old die had to be used.
The US President Jackson presented the Sultan with the dollar as a present, thanking him for the ratification of a new trade treaty.
It was sold by Bowers and Merena Inc of Wolfeboro, New Hampshire. The previous record, of $1.8m (£1.2m), was set by the same auctioneers two years ago for another of the set of eight, which had been struck as a gift for the King of Siam.
Dollar sets record for a US coin
US: A NEW record for a United States coin has been achieved at auction in New York with the sale of a silver dollar for a premium-inclusive $4.14m (£2.63m) on August 30.
