The origin of golf has long been debated. Some historians trace the sport back to the Roman game of paganica, which involved using a bent stick to hit a wool- or feather-stuffed leather ball. According to one view, paganica spread throughout several countries as the Romans conquered much of Europe during the 1st century bc and eventually evolved into the modern game. Others cite chuiwan (ch’ui-wan) as the progenitor, a game played in China during the Ming dynasty (1368–1644) and earlier and described as “a game in which you hit a ball with a stick while walking.” Chuiwan is thought to have been introduced into Europe by traders during the Middle Ages. However, upon close examination, neither theory is convincing."

Source Encyclopædia Britannica

It seems likely that there may be links to another ancient ball game called Kolf :

Kolven (verb; or noun: kolf) is a game originated in the Netherlands, played by four people in which players hit the ball over a certain distance and the first people to reach their opponents' starting point wins. The origins of Kolf are also unclear, but records show that it existed as far back as the 13th century.

More info on Kolf at Wikipedia