The company will also allow publishers to charge AI crawlers for access using a new “pay per crawl” model