The St George's Street building, in Semilong, is to be replaced with a new facility, which is currently being built in St James.
Sikh leaders have now submitted a planning application asking for the go-ahead to create 17 bedrooms in the current temple, over four floors.
But the Northampton Town Centre Conservation Advisory Group, an independent organisation, said the plans might cause annoyance to neighbours.
Chair Sue Biddle said: "[There are] concerns regarding the building impact of HIMOs with the increase of traffic flow, car parking, rubbish collection and fly tipping, and the social stability of the area.
"We recommend refusal of this application."
The architects' drawings for the project take in the ground floor, first floor, second floor and basement.
Environmental Health officer Matt Clough said he could not support the plan for eight separate reasons, mainly to do with fire safety and cramped conditions.
Drawings suggest that the basement bedrooms would have no windows that can be opened, and areas of the second floor would have ceiling heights of less than two metres.
Commenting on the basement bedrooms, he added: "Both basement bedrooms are inner rooms to the kitchen, living room and dining room.
"Scrutiny of the block plan appears to show the internal courtyard connects only with the premises at 17a St Georges Street. It therefore does not lead to a place of safety."
The move to a new £2 million temple, in St James Mill Road, was set in motion after Sikh leaders decided there were too few parking spaces for large weddings, funerals or celebrations.