The muon anomalous magnetic moments (g-2) have been measured (BNLE821) and calculated to a precision (0.54 ppm), that it may be used to probe new physics at a much higher energy scale not reachable by the present-day accelerators. The on-going experiment at Fermilab (E989) and the future J-PARC experiment (E34) are aiming to reduce the experimental uncertainty by a factor of four. The lattice calculation of the hadronic light-by-light contribution to muon g-2 is crucial to reduce the theoretical uncertainty to a similar level. We have already calculated the connected diagrams and the leading disconnected diagrams using the RBC/UKQCD $48^3 \times 96$ ensemble with physical pion mass and 5.5 fm box size. I will report our recent progress on taking the continuum and infinite volume limits, which are the two major systematics in our previous calculation.
