If you can do an earlier breakfast, I highly recommend The Cottage:
http://www.cottagelajolla.com/
They have a nice outdoor patio and the food's really good, but it does get really busy on the weekends and often has a long wait if you don't get there early.
One of our favorite sandwich spots is Board N Brew, which is not too far north of La Jolla:
http://boardnbrewdelmar.com/
You can grab stuff for lunch and eat at Powerhouse Park in Del Mar, gorgeous views and plenty of grass for relaxing and catching some sun.
Definitely head down to the Children's Pool to see the sea lions, and there's a little park/playground area at La Jolla Shores that would be fun (although, since your kiddo is so young they may not get much out of it this time around)
Have a fantastic anniversary!