Module "isomorphic-packages/sitemap-crawler/src/findByRobots"

findByRobots

• findByRobots(url: string, forceSitemapArgument?: string): Promise<string | false>

• Look for a valid robots.txt and parse Sitemap url.

  see

  https://regex101.com/r/Trn97e/1

  Parameters

  • url: string

  • Optional forceSitemapArgument: string

    Append this query argument to the robots.txt in case you can server-side activate the sitemap

  Returns Promise<string | false>