ChatGPT Fails Catechism Class!
I learned something interesting about the way ChatGPT processes information that might help you get better results when using AI tools.
A friend asked ChatGPT a straightforward question: “Does St. Patrick’s Day always fall within Lent?”
Simple enough, right? You’d expect a clear yes or no answer. But ChatGPT gave a completely wrong and nonsensical response. My friend assumed he might not have worded his question correctly, so he tried again. Still wrong. Even comically so.
That got me thinking about a key limitation of AI. ChatGPT is, first and foremost, a language model. It can do math, but it doesn’t always recognize when it should be doing math. Sometimes, instead of treating a question as a math problem, it defaults to pattern-matching text, leading to incorrect results.
Why Did ChatGPT Get It Wrong?
The answer to my friend’s St. Patrick’s Day question requires comparing two sets of dates:
- Lent runs from Ash Wednesday to Easter.
- St. Patrick’s Day is always March 17th.
This is a math problem. First you have to determine the possible dates of Easter, which moves around every year. Then you have to calculate when Ash Wednesday would occur, based on the date of Easter. Then you have to check whether March 17th falls within that range.
Instead of running a calculation, ChatGPT seemed to approach my friend’s question as a text-based comparison, which led to an incorrect answer.
How to Get Better Answers from AI
When using ChatGPT (or any AI tool), it’s important to know when your question requires a mathematical or logical calculation rather than just text-based or pattern-matching reasoning. If you need an AI to do math, be explicit about it.
- Tell it to calculate. Use clear instructions like “Use math to determine the answer.”
- Give it structured data. If you provide specific dates and ask for comparisons, it may be more likely to use the correct reasoning process.
- Ask the AI to check its own answer. This seems strange, but sometimes you can ask “is the answer you just gave me correct,” and sometimes ChatGPT will discover its own mistake.
- Double-check AI-generated answers. If something seems off, verify the response with another source.
Final Thoughts
I use ChatGPT all the time. It’s a powerful tool and it helps me with many tasks. But it is just a tool, and it has limitations.
Having a basic understanding of how AI processes information can help you avoid misleading answers and use AI tools more effectively.
When you ask an AI to step outside its “comfort zone” — like moving from language analysis to calculations — make sure to give it clear instructions to that effect.
If you’re curious, here’s a good page on The Date of Easter. The simple answer is that Easter occurs on the first Sunday after the first full moon after the vernal equinox.
