Oskar Groening was sentenced to four years for his role as an accessory to the murder of 300,000 of the Auschwitz concentration camp's roughly 1 million victims.
The district court of Lueneburg, in 2015, sentenced Groening to four years' imprisonment for aiding murder during the Holocaust.
Groening died on Friday at the age of 96, said the Spiegel Online.
German prosecutors deemed Groening to be fit enough to go to prison as long as there was appropriate medical care.
In December 2017, the Federal Constitutional Court of Germany decided that Groening was liable and his punishment must begin. He submitted a request for clemency in early March.
The start of Groening's prison sentence was delayed by legal action and ill health.
He said he witnessed individual atrocities, but did not acknowledge participating in any crimes.
Zuroff pointed out that two other Nazi war criminals - John Demjanjuk, who was convicted in 2011, and Reinhold Hanning, who was convicted in 2016 - also died while awaiting decisions on their appeals.
"Without at least symbolic justice, these trials, as important as they are, lose an important part of their significance", said the Simon Wiesenthal Center, a Jewish human rights organization that has worked to track down the world's remaining Nazis. The man known as the "Bookkeeper of Auschwitz" was convicted in July 2015.
Although his lawyers argued he did not kill anyone during his time in Auschwitz.