Random thoughts
I may contradict myself by the end or maybe there was no point to make.
There is a plethora of fields where AI is making progress or at least, envisaged to solve problems with the technology getting better. People are good at cutting the search space by a billion-fold. People with AI can multiply the effect. But is that true?
It is possible to reduce the search space trillion-fold by having the AI accelerate the search by going and verifying potential paths. But for a moment, let's view the AI in isolation. They are next token predictors. Language models produce the next word, not just any word, but a specific word out of all the possible words in the English dictionary (including producing no word at all). Predicting the next word is a way of cutting the search space. When LLMs produce the next token, what is going to come after and till the end is already specified in its distribution. The distribution changes every time the model produces a token. The parameters, such as greedy, top-p, temperature, etc., just change how you draw from the distribution, but they can't be outside the distribution. Large Language Model doesn't know what it knows and doesn't know what it doesn't know.
It's obvious to see that predictions in LLMs are different from those of humans. Human prediction is more complex than the next word. Humans can make predictions on longer timescales, not just what the next sentence will be. Whether the prediction holds is a different matter (long-term predictions tend to be less reliable in general). This makes a different kind of search problem. Humans can cut the search space with a better understanding of the problem and sniping on important events, not autoregressively.
Some people will point to scientific results from fields outside their own that are marketed as having been done with "AI" and ask: How could you take a hard line against "AI" when it has provided XYZ? Such remarks both conflate many different things under the umbrella of "AI" and are also usually based on hype-filled media coverage that tends to obscure what technology was actually used in what way. My hypothesis on the use of AI in open-ended research is that it's not fulfilling. Current AI can't come up with better explanations or thoughtful experiments, so one would use them to explore ideas where a rival theory already exists. It can still be a powerful tool for generating conjectures or speeding up the boring parts so humans can iterate faster.