I know "Casual Friday" has been done to excess, but it can be worn with style.
Whether at home or the office or elsewhere, you can make casual attractive. Here's an example: today I had to garden, run errands, bake and cook a decent dinner (too many sandwiches lately!) and I wanted to be comfortable, but wasn't about to wear sweats. (I do have them. I use them for yoga or running. Period.)
Today I wore vintage dungarees (ca. 1932...they belonged to my Great Uncle Jack, who wore them in the Navy) a corn yellow "Three Hour Sweater", which I knitted myself a few years ago. The pattern was available free online at the time I made the sweater. The pattern itself is from the 1930s. It's got a ribbed neck area, puffed sleeves and a ribbed waist. Simple and comfortable, but tailored. To that I added argyle socks (cream, with pale leaf green, peach & yellow), brown loafers, a Bakelite bangle and Bakelite earrings.
If you'd like to see some cute examples from the 30s and 40s, check out any movie star biography from that time period. See Ginger Rogers after a successful fishing trip, or other actresses and starlets posing for the old movie magazines. Loose dungarees with a proper high waist, cuffed or rolled, anklets or argyles, loafers, and a bright, tailored shirt and you're good to go. Add coordinating Bakelite or celluloid jewelry and you're all set.
Set your hair or roll it (or use a do-rag or pony tail) and someone is sure to walk up to you and say,"My great aunt dressed just like that when....".