Auto-translated
The first mistake was about the pressure being lower at the bottom. Let's correct it:
mg = integrate (h from 0 to H) ρg(H-h)*dS(h) = integrate (h from 0 to H) ρgH*dS(h) - integrate (h from 0 to H) ρgh*dS(h) ------------- (1)
The second mistake was forgetting R in one place when expanding the small area element dS, and then losing another R somewhere:
integrate (h from 0 to H) ρgh*dS(h) = integrate (α from 0 to arcsin(H/R)) ρgRsin(α)*2π(Rcos(α))*Rdα = integrate (α from 0 to arcsin(H/R)) ρgR*π*sin(2α)*R^2*dα = -0.5πρgR^3cos(2α) (α from 0 to arcsin(H/R)) = -0.5πρgR^3 ( cos(2arcsin(H/R))-1) = -0.5πρgR^3 ( 1-2(H/R)^2-1) = πρgRH^2 -------------- (2)
The first integral from the sum in (1):
integrate (h from 0 to H) ρgH*dS(h) = integrate (α from 0 to arcsin(H/R)) ρgH*2π(Rcos(α))R*dα = 2πρgHR^2 * integrate (α from 0 to arcsin(H/R)) cos(α)dα = 2πρgHR^2 * sin(α) (α from 0 to arcsin(H/R)) = 2πρgHR^2 * H/R = 2πρgRH^2 (3)
Let's combine (1), (2), and (3), and remember to cancel out g from both parts, which I forgot to do last time.
m = 2πρRH^2 - πρRH^2 = πρRH^2
H = sqrt(m/πρR)
If needed, I won't ask for another attempt, as there's nothing more to do for now.