Options
All
  • Public
  • Public/Protected
  • All
Menu

Module "isomorphic-packages/sitemap-crawler/src/findByRobots"

Index

Variables

Functions

Variables

Const ROBOTS_TXT

ROBOTS_TXT: "robots.txt" = "robots.txt"

Functions

findByRobots

  • findByRobots(url: string, forceSitemapArgument?: string): Promise<string | false>
  • Look for a valid robots.txt and parse Sitemap url.

    see

    https://regex101.com/r/Trn97e/1

    Parameters

    • url: string
    • Optional forceSitemapArgument: string

      Append this query argument to the robots.txt in case you can server-side activate the sitemap

    Returns Promise<string | false>

Generated using TypeDoc