Brown was the son of Gawen and Elizabeth (Byles) Brown, and descended from the Rev. Increase Mather on his mother's side. He was taught by his aunt and around 1773 (age 12) became a pupil of Gilbert Stuart. He arrived in London in 1781 to further his training in Benjamin West's studio, entered the Royal Academy schools in 1782 with plans to be a miniature painter, and began to exhibit a year later.
In 1784, he painted two religious paintings for the church of St. Mary's-in-the-Strand, which led Brown to found a partnership with the painter Daniel Orme for the commercialization of these and other works through exhibition and the sale of engravings. Among these were large paintings of scenes from English history, as well as scenes from Shakespeare's plays. However, despite their success he began to concentrate on portraiture. His first successes were with American sitters, among others his patron John Adams and family in 1784–85; this painting is now in the Boston Athenæum. In the spring of 1786, he began painting the earliest known portrait of Thomas Jefferson, who was visiting London. He also painted Charles Bulfinch the same year. He was elected a Foreign Honorary Member of the American Academy of Arts and Sciences in 1798.[1]
A falling off patronage in the mid-1790s, and failure to be elected to the Royal Academy, led Brown to leave London in 1808 for Bath, Bristol, and Liverpool. He settled in Manchester, returning to London almost two decades later, in 1824, where, even after West's death, he continued to imitate his teacher's style of painting. Unable to secure commissions, Brown eventually died in poverty in London.