Francis was the second son, and one of twelve children of Dr. Thomas (1715–1794) and Mildred Walker. His mother was the widow of Nicholas Meriwhether and through her he inherited the estate at Castle Hill in Albemarle County. His older brother John represented Virginia in the Continental Congress and the U.S. Senate.
Francis married Jane Byrd Nelson, daughter of General Hugh Nelson, and granddaughter William Nelson who had been President of the council and acting governor of Colonial Virginia. They had two daughters: Jane and Judith. Judith married William C. Rives who was later a U.S. Senator for Virginia.
Besides his state and federal service, Walker was a judge in Albemarle County, and Colonel of the county's militia. He died at home in 1806 and was buried in a family plot on his estate of Castle Hill.
Castle Hill (Virginia) still stands and is on the National Register of Historic Places. It is off Virginia Highway 231, north of Interstate 64 and northeast of the community of Cismont in Albemarle County. The property was sold in 2003 for 24 million (U.S. dollars). There is a roadside historic marker, but the estate itself is private property.
