Asking what day is today... surprise

I could swear I posted something similar in the past… but I can’t find it :frowning:

LLMs are notoriously bad at two things:

  • recognizing when they don’t ACTUALLY have the answer to a question
  • counting the number of "r"s in the word “strawberry”

Along those lines, if the LLM doesn’t have the targeted day of week, it will occasionally decide that it in fact DOES know the targeted day of the week, and it will have a 1 in 7 chance of getting it right :frowning:

“We” (people who create AI things in general) generally get around this by nudging the LLM in the right direction (give it a date calculator tool, give it the current full date, etc)… but it’s not foolproof.

TLDR; I’m going to tag @simon and ask him to raise it as a hard to reproduce bug :slight_smile:

We’re working on some unrelated updates at the moment that will give us a great place to fix this… but it will be measured in weeks/months… so I’m hoping we can make it easier to reproduce/fix before then.

1 Like