All golf course design projects are unique from one to another and each has their own variables and client needs / desires. So there is no formula or guide that I’m aware of that you can work through and come up with a design fee proposal.
My experience has been that the design team will typically receive 5-7% of the construction costs of the project (in the United States). To arrive at a construction cost, I would define the client’s needs and create a concept / preliminary golf course design. Do a take-off based on the design to determine quantities, areas, etc. and develop a cost estimate that includes all work that will be done such as grading, drainage, irrigation, seeding, sodding, restrooms, putting green soil mix, bunker sand, pond / lake construction etc…etc.. Also keep in mind if you will be project managing the club house construction, parking lot, and like items or if the client prefers to have these handled through an A/E firm.
So bottom line, I would create a concept design of the course layout and develop a construction cost estimate based on the concept plan and multiply the total construction cost by 5-7% (In the U.S.) This would get you an idea of the design fee to propose to the client.