No Sweets = Extra Fat Loss?

I haven't been working out or running a whole lot these last couple of weeks, but I have been eating pretty clean and eating no sugar (really just no sweets like ice cream, candy...etc.) and I have lost 2.8 lbs!  I think that's proof that eating clean is a bigger part of getting lean than working out.  I think both are important, but there are a lot of people that think that if they run or work out a lot, they can eat whatever they want.  That might be true for some people who are already in great shape, but that's no way to lose fat!