#Any search crawler can crawl our site User-agent: * #Allow only below mentioned paths #Allow: /en/ #Allow: /fr/ #Allow: /gb/ #Allow: /in/ #Disallow everything else #Disallow: /