Mmm, but wealth isn't cash money - it's "stuff people want". It's entirely a human concept. If there is more "stuff people want" being produced then the economy is growing, it is just growing in a different direction. That the desired commodity is changing doesn't alter this. If people want longer lives, or more leisure time, or whatever, instead of more consumables, that all has economic consequences. The de-growth movement, a lot of it is shifting from many, cheap, mass-produced items to fewer, expensive, hand-made items.
Wealth is "tangible stuff people want". There is a balancing point where working less for the same amount of stuff becomes more important than getting more stuff. EX: Suppose you are working 80 hour weeks and making 200k in 5 years would you rather work 40 hour weeks and make 200k or 80 hour weeks and make 400k?
That is not to say working less is always a major goal, if someone was doing 20 hour work weeks for 30k they would probably rather have 20 hour work weeks that pay 60k than 10 hour work weeks that pay 20k. The great thing about free markets is they let people balance many of those less tangible goals. However, government policy can easily focus on progress that is out of whack with what people actually want so an understanding that the Chinese population might tolerate more pollution for more growth were an American population cares more about clean air than maximizing growth is nessesary to keep people happy.