Core: fixed potential buffer overrun when initializing hash.

Initial size as calculated from the number of elements may be bigger
than max_size.  If this happens, make sure to set size to max_size.

Reported by Chris West.