Stourbridge College owns the garden which is part of its horticultural training centre at the park, off Mucklow Hill.
The centre, which features the grade one listed walled garden as well as mobile buildings, a greenhouse and a cafe, is set to close in August.
The Friends of the Leasowes said the centre, which was owned by Dudley Council, until it was passed to the college 23 years ago, should return to the council’s ownership.
Chairman Marianne Diller said: “We are still hoping the council will insist it comes back into its care.
“We have lobbied various people and are hoping they will be able to persuade Stourbridge College that it belongs to the people of Dudley and should come back to them and become part of the park again.
“It is a crucial part of the park and could fall subject to vandalism if it’s left empty - there is an opportunity for it to become an asset for the community.”
But Dudley Council would not say if it was prepared to take over the centre.
John Millar, Dudley Council’s director of urban environment, said: "We have been informed that the college is planning to move out of its location within the grade I listed Landscape Heritage Area at The Leasowes.
“We will continue to work with them and the friends group to look at the future of the site."
Around 60 students with learning disabilities are taught at the college’s Leasowes Park site which covers around two acres.
Courses taught there will transfer to Stourbridge College’s new Kidderminster site - Chain Wire Sports Club - which it has bought for £700k from owner Brintons carpet manufacturers.
The college has said it is undertaking a feasibility study on the site with regards to its future use.