# Dynamic robots.txt & sitemap.xml Setup Guide

This guide explains how to use the dynamic robots.txt and sitemap.xml generation system for your multi-tenant stores.

## 🎯 Overview

The system provides:
1. **Dynamic robots.txt** - Generated on-the-fly for each tenant with their specific domain
2. **Sitemap.xml generation** - Command to generate sitemaps for all stores
3. **Automated scheduling** - Cron job support for daily sitemap updates

## 📁 Files Created

### 1. Controllers
- `packages/Webkul/Shop/src/Http/Controllers/SEOController.php`
  - Handles dynamic robots.txt generation
  - Serves sitemap.xml from storage

### 2. Console Commands
- `packages/Webkul/Shop/src/Console/Commands/GenerateSitemap.php`
  - Generates sitemap.xml for all active tenants
  - Supports single company generation

### 3. Routes
Added to `packages/Webkul/Shop/src/Http/routes.php`:
- `/robots.txt` → SEOController@robots
- `/sitemap.xml` → SEOController@sitemap

### 4. Service Provider
Updated `packages/Webkul/Shop/src/Providers/ShopServiceProvider.php`:
- Registered the GenerateSitemap console command

## 🚀 Usage

### Testing robots.txt

Visit any tenant store domain:
```
https://bazaarksa.shop/robots.txt
https://yourstore.com/robots.txt
```

The robots.txt will be generated dynamically with:
- The correct domain in the Sitemap URL
- All allow/disallow rules as specified
- No caching issues

### Generating Sitemaps

#### Generate for ALL active companies:
```bash
php artisan sitemap:generate
```

#### Generate for a SPECIFIC company:
```bash
php artisan sitemap:generate --company_id=123
```

### Where Sitemaps are Stored

Sitemaps are saved to:
```
public/storage/sitemaps/{domain}_sitemap.xml
```

Example:
```
public/storage/sitemaps/bazaarksa_shop_sitemap.xml
public/storage/sitemaps/yourstore_com_sitemap.xml
```

### Accessing Sitemaps

Visit any tenant store domain:
```
https://bazaarksa.shop/sitemap.xml
https://yourstore.com/sitemap.xml
```

The system will:
1. Check if a cached sitemap exists in storage
2. Serve the cached version if found
3. Generate a basic sitemap if not found

## ⏰ Setting Up Cron Job (2 AM Daily)

### For Windows (Task Scheduler)

1. Open Task Scheduler
2. Create a new task:
   - **Name**: Generate Sitemaps
   - **Trigger**: Daily at 2:00 AM
   - **Action**: Start a program
   - **Program**: `C:\xampp\php\php.exe`
   - **Arguments**: `C:\xampp\htdocs\glob\Code\html2\artisan sitemap:generate`
   - **Start in**: `C:\xampp\htdocs\glob\Code\html2`

### For Linux (Crontab)

Add to crontab:
```bash
crontab -e
```

Add this line:
```
0 2 * * * cd /path/to/your/project && php artisan sitemap:generate >> /dev/null 2>&1
```

### For Laravel Scheduler (Recommended)

Add to `app/Console/Kernel.php`:
```php
protected function schedule(Schedule $schedule)
{
    $schedule->command('sitemap:generate')
             ->dailyAt('02:00')
             ->withoutOverlapping();
}
```

Then set up a single cron entry:
```
* * * * * cd /path/to/your/project && php artisan schedule:run >> /dev/null 2>&1
```

## 📊 What's Included in Sitemap

For each tenant store, the sitemap includes:

1. **Homepage** (Priority: 1.0, Daily)
   - Example: `https://bazaarksa.shop/`

2. **CMS Pages** (Priority: 0.8, Weekly)
   - Example: `https://bazaarksa.shop/page/about-us`
   - Example: `https://bazaarksa.shop/page/contact`

3. **Categories** (Priority: 0.7, Weekly)
   - Example: `https://bazaarksa.shop/categories/25398`

4. **Products** (Priority: 0.9, Weekly)
   - Example: `https://bazaarksa.shop/product-details/281535`

## 🔧 Customization

### Modify robots.txt Rules

Edit the `generateRobotsTxt()` method in:
```
packages/Webkul/Shop/src/Http/Controllers/SEOController.php
```

### Modify Sitemap URLs

Edit the URL generation methods in:
```
packages/Webkul/Shop/src/Console/Commands/GenerateSitemap.php
```

Methods to customize:
- `addCMSPages()` - CMS page URLs
- `addCategories()` - Category URLs
- `addProducts()` - Product URLs

### Change Sitemap Storage Location

Edit the `saveSitemap()` method to change where sitemaps are stored.

## 🌐 CDN Integration (Optional)

If you want to serve sitemaps from CDN (cdn2.markatty.com):

### Option 1: Upload to CDN After Generation

Modify `saveSitemap()` method in `GenerateSitemap.php`:

```php
protected function saveSitemap($company, $xml)
{
    // Save locally first
    $storageDir = public_path('storage/sitemaps');
    if (!file_exists($storageDir)) {
        mkdir($storageDir, 0755, true);
    }

    $domain = $company->domain ?: $company->cname;
    $filename = str_replace('.', '_', $domain) . '_sitemap.xml';
    $filepath = $storageDir . '/' . $filename;

    file_put_contents($filepath, $xml);

    // Upload to CDN
    $this->uploadToCDN($filepath, $filename);
}

protected function uploadToCDN($filepath, $filename)
{
    // Add your CDN upload logic here
    // Example: Use FTP, S3, or HTTP API to upload
}
```

### Option 2: Serve from CDN in SEOController

Modify `sitemap()` method in `SEOController.php`:

```php
public function sitemap()
{
    $domain = request()->getHost();
    $filename = str_replace('.', '_', $domain) . '_sitemap.xml';
    
    // Try to fetch from CDN
    $cdnUrl = 'https://cdn2.markatty.com/sitemaps/' . $filename;
    
    try {
        $content = file_get_contents($cdnUrl);
        return response($content, 200)
            ->header('Content-Type', 'application/xml');
    } catch (\Exception $e) {
        // Fallback to local storage
        $sitemapPath = public_path('storage/sitemaps/' . $filename);
        
        if (file_exists($sitemapPath)) {
            $content = file_get_contents($sitemapPath);
            return response($content, 200)
                ->header('Content-Type', 'application/xml');
        }
    }

    return response($this->generateBasicSitemap(), 200)
        ->header('Content-Type', 'application/xml');
}
```

## 🧪 Testing

### Test robots.txt:
```bash
curl https://bazaarksa.shop/robots.txt
```

### Test sitemap.xml:
```bash
curl https://bazaarksa.shop/sitemap.xml
```

### Test command:
```bash
php artisan sitemap:generate --company_id=9482
```

## 📝 robots.txt Rules Explained

### ✅ Allowed (SEO-friendly):
- Homepage: `/`
- CMS pages: `/pages/page/*`
- Categories: `/pages/categories/*`
- Product details: `/pages/product-details/*`
- Questionnaire: `/pages/questionnaire/*`
- Contact page: `/pages/page/account/contact`

### 🚫 Disallowed (Non-SEO):
- Account pages (login, register, dashboard, etc.)
- Cart and checkout pages
- Admin and API routes
- Duplicate product layouts
- Internal pages (search, compare, etc.)

## 🐛 Troubleshooting

### Issue: Sitemap not found
**Solution**: Run `php artisan sitemap:generate` to create sitemaps

### Issue: robots.txt shows wrong domain
**Solution**: Check your tenant domain configuration in the database

### Issue: Command not found
**Solution**: Run `php artisan config:clear` and `composer dump-autoload`

### Issue: Permission denied when saving sitemap
**Solution**: Ensure `public/storage/sitemaps` directory has write permissions:
```bash
chmod -R 775 public/storage/sitemaps
```

## 📈 Performance Notes

- Sitemaps are generated once and cached
- robots.txt is lightweight and generated on-the-fly
- Large stores (10,000+ products) may take 1-2 minutes to generate
- Consider using chunking for very large product catalogs (already implemented)

## 🔐 Security

- No sensitive data is exposed in robots.txt or sitemap.xml
- Only public, SEO-friendly URLs are included
- Account pages and admin routes are properly disallowed

## 📞 Support

For issues or questions, check:
1. Laravel logs: `storage/logs/laravel.log`
2. Command output when running manually
3. Verify database connections for tenant data

---

**Last Updated**: 2025-10-30
**Version**: 1.0