Would it burn my bridge to quit a summer job at the end of the summer if I implied that I would not quit?

I got a job at a retail store at the beginning of the summer and I implied that I would keep working during the year next year which I did intend on doing at the time. I have thought about it a lot and I just don't think it would be a good idea to keep my job during the year, I'm not crazy about the job and I don't want it to get in the way of my grades and I would rather get involved in other extra curricular activities ( I will be a sophomore in college). The main reason I got a job in the first place is too get experience and references when I apply to other internships and jobs in the future. Would it prevent me from using this job as a reference in the future if I quit the job after only 3 months? I was thinking I would use an excuse like my schedule got changed at the last minute or something, which may actually happen. I feel like if I quit and end up burning my bridge with them the job will have been a waste of time because even though the money is nice, I didn't really need it that much.