Speed problem of terrain objects

Now the terrain tiles are objects and everything is fine, right? No. The map displaying is somewhat slower from average 5-10ms up to 40ms per step (in small rooms it's under 5ms though). It's not that bad I guess and optimizing gives more speed.

Then I noticed something strange. When I quit the game there is a noticeably long pause. And it becomes worse when you visit more levels. For some reason deleting objects from maps is incredibly slow. I guess it can be, because there are 10 000 tile objects in a 100 x 100 map. Still it's surprisingly slow, taking several seconds with just 4-5 maps. Imagine you visit all levels in the game and wait for 5 minutes for it to delete the objects when you leave the program.

Is delete really that slow or am I doing something wrong? There isn't a big difference in std::vector compared to raw array of pointers (which is easy to test, because the syntax is the same). Raw array seems to be a little bit faster, but not in a way that would make a real difference.

So, what you can do in a situation like this? Should I return to the model where only one level is stored in the memory at a time, loading the level data from files when changing the level? Maybe, if nothing else works. The class hierarchy for game objects is quite heavy, at least now when Tile object of a map could in fact own/contain many of the static object types which don't move. That would remove the "double" pointers for objects, one in the list and other in a form of quick pointer in a Tile. The entire model could be switched from list to a type where Tile owns all objects.

The base class for objects could be also simplified and some functionality moved forward in the inheritance level possibly to intermediate classes (which then would make the class hierarchy more complex and who knows what happens then).

The map itself could contain Tile objects only in the visible areas and empty pointers in solid rock areas of the level.

Or maybe this problem will go away by itself in 10 years when computers become faster.