One thing that can be said about AI chatbots, whether Google Bard or ChatGPT, is that they can resort to scraping the Internet to answer your questions. Perhaps to thwart complaints and potential lawsuits, Google is now giving web publishers the option to hide from Bard's gaze.

Google has announced that there is a new control in the robots.txt index file called Google Extended. This file is intended to let Google know which URLs are accessible to Google's crawlers. This new control means that publishers can decide if their content will "help improve the Bard and Vertex AI generative APIs (including the future generation models that power these products)."

According to Google's announcement, this is being done as a result of feedback from web publishers who want to choose and control how they use content related to generative AI. Yet the term itself is all about how access helps AI improve over time. It's as if Google is trying to suggest that keeping Bard (and its cloud-based machine learning platform, Vertex) away from your work is holding it back.

Generative AI has become a concern for some publishers, especially as companies like Google further integrate those services into products like search. After all, every time a piece of information you create is used in a chatbot conversation, whether it is cited or not, it means one less person coming to your site for the same information.

Fewer visitors inevitably means less revenue from advertising and whatever other sources of income the site depends on to keep it running. Of course, other publishers may see other benefits in having their content appear in chatbot responses, and as a result may be happy to have Bird crawl their content.

So it is about choice that matters, because what generative AI needs most right now is more controversy. Furthermore, if your service relies on users being able to access and redistribute information whenever they request it, the last thing you want to do is upset the people who provide it.

Nevertheless, as a user, you may not be able to use Google Bard as much as you would like in the near future. Google will undoubtedly adapt to the loss of content from publishers who choose to opt out, and with so much content on the Internet, it should be possible to get answers to questions. However, it will be interesting to see how much this change will affect Bird's knowledge in the future.