He made the comments in a discussion about making improvements to his Oxfordshire farm.
Jeremy Clarkson said he “simply can’t get planning permission” and said local planners are influenced by “people in the village who wear red trousers”.
The broadcaster, 62, said “no” is the council’s answer to “everything” when it comes to his property, named Diddly Squat farm.
In an interview with TalkTV’s The News Desk, Clarkson said: “Maybe I should buy an apron and join the Masons.
“I don’t know what you have to do, but I simply can’t get planning permission for anything, which is infuriating, but it’s not just me as it turns out.
“I thought it was, but farmers up and down the country are saying the same thing.”
The farm, subject to an Amazon Studios series called Clarkson’s Farm, has proved popular with visitors ever since the show was broadcast last June.
Clarkson bought the farm in 2008 and it was run by a villager but, when he retired in 2019, the TV presenter decided to see if he could run it himself.
The success of the series has seen people flock to the farm shop to buy products such as Cow Juice, rapeseed oil, chutneys and jams.
Reports have previously said neighbours had been left annoyed by the amount of shoppers who have queued for hours to purchase goods.
Talking about planning, Clarkson told the programme: “Without knowing it West Oxfordshire District Council is writing a fantastic script and every farmer in the country will go ‘That’s exactly what’s happening’.
“You know, these, how can I put it, not terribly bright people in planning departments, just don’t understand what they’re messing around with.
“And I’m seeing the results. I was told to change the traditional green tin roof on my shop to much more expensive slate.
“I was told I couldn’t sell milk that was coming from five miles away from a woman who’s desperately, desperately worried about her future as a dairy farmer because of TB and so on.
“I haven’t been allowed to build a farm track, I haven’t been allowed to build a car park even though the locals are saying there’s too many people parking on the road.
“It just goes on and on and on and the council’s answer to everything is ‘no’.”
Asked about what he wants the Government to do to help farmers, Clarkson said: “Well, yes, I think there is a role for the Government.
“I think at the moment, farmers as I understand it – and I am a trainee farmer let’s make no mistake about that – are allowed to change buildings that are smaller than 150 square metres, which is very, very small.”
He said he thinks it should be slightly larger, adding: “I think that farmers shouldn’t be allowed to build solar farms or housing estates without proper local consultation obviously.
“But I just think that the government should enable farmers to alter buildings of, say 500 square metres, without necessarily having to go to local planners, who are inevitably swayed by people in the village who wear red trousers and make fools of themselves and object.
“So that’s what I would like to see, it’s just a little bit more.
“If they’re going to say to farmers ‘You must diversify’, they must say to local authorities ‘And you’ve got to let them’.”
In March, Clarkson reapplied for planning permission for a car park extension on his farm.
The proposal was for an “extension to existing parking area to formalise temporary parking and provision of new access arrangements” at the Diddly Squat Farm Shop.
The proposal also included a “new storage compound and associated landscaping”.
It was rejected this month by West Oxfordshire District Council, which gave its refusal reasons in its notice of decision by saying that due to its location, size and design the proposed development would “not be sustainable and would not be compatible or consistent in scale with the existing farming business or its open countryside location and would have a visually intrusive and harmful impact on the rural character, scenic beauty and tranquillity of the Cotswolds Area of Outstanding Natural Beauty”.