Как да използвам MAX() за резултат от подзаявка?

Не ви е необходима подзаявката, която намира максималната стойност.
Вместо това,; имате нужда само от първото ред, след като сте поръчали редовете:

select * from (
  select 
    membership.mem_desc,
    membership.mem_max_rentals,
    membership_history.mem_type,      
    count(membership_history.MEM_TYPE) as membership_count
  from membership_history
  JOIN membership ON membership.mem_type = membership_history.mem_type
  group by (membership_history.mem_type,membership.mem_desc,membership.mem_max_rentals)
  ORDER BY 4 DESC  -- Added this line
) g
WHERE ROWNUM = 1. -- Added this line