TOKEN SYSTEM REDESIGN

File: TOKEN_SYSTEM_REDESIGN.md

Loading documentation...

# Token System Redesign Documentation

## Overview

The token system has been redesigned to support:
1. **Detailed token usage logging** - Track every token usage event
2. **Transcribe tokens** - New token type for transcription minutes
3. **Monthly token allocations** - Automatic token allocation from subscription plans
4. **Token rollover** - Unused monthly tokens carry over to next month
5. **Enhanced reporting** - Detailed usage analytics and cost analysis

## New Database Tables

### 1. token_usage_logs
Tracks individual token usage events with detailed context.

```sql
- id (primary key)
- user_id (foreign key to users)
- token_purchase_id (foreign key to token_purchases, nullable)
- feature_type (string) - e.g., 'sms_tokens', 'email_tokens', 'transcribe_tokens'
- amount_used (integer) - Number of tokens used in this event
- usage_context (string, nullable) - e.g., 'sms_send', 'email_send', 'transcribe_audio'
- reference_type (string, nullable) - e.g., 'sms', 'email', 'audio_file'
- reference_id (bigint, nullable) - ID of the related record
- metadata (json, nullable) - Additional context data
- used_at (timestamp) - When the tokens were used
- created_at, updated_at (timestamps)
```

### 2. Enhanced token_purchases
Added fields for monthly allocations:

```sql
- is_monthly_allocation (boolean) - Whether this is a monthly allocation from a plan
- allocation_month (string) - The month this allocation is for (YYYY-MM format)
- plan_id (foreign key to plans, nullable) - Reference to the plan that provided this allocation
```

## New Token Types

### Transcribe Tokens
- **Purpose**: Track transcription minutes for audio/video files
- **Units**: Minutes of audio/video content
- **Packages Available**:
  - 60 Minutes: $12.00
  - 300 Minutes: $50.00
  - 600 Minutes: $90.00

## Monthly Token Allocations

### How It Works
1. **Automatic Allocation**: Each month, users with active subscriptions receive token allocations based on their plan
2. **Plan-Based Limits**: Token amounts are defined in plan features
3. **Rollover Support**: Unused tokens from one month carry over to the next
4. **Expiration**: Monthly allocations expire at the end of the month

### Plan Features
Plans now include specific token limits:

**Basic Plan**:
- SMS Tokens: 100/month
- Email Tokens: 500/month
- Transcribe Tokens: 60 minutes/month

**Pro Plan**:
- SMS Tokens: 500/month
- Email Tokens: 2000/month
- Transcribe Tokens: 300 minutes/month

## Usage Flow

### 1. Token Usage with Logging
```php
// Use tokens with detailed logging
$usageService->useTokens(
    user: $user,
    featureType: 'sms_tokens',
    amount: 1,
    usageContext: 'sms_send',
    referenceType: 'sms',
    referenceId: $smsId,
    metadata: ['recipient' => '+1234567890', 'message_length' => 140]
);
```

### 2. Token Priority (FIFO)
1. **Plan Limits** - Use from monthly plan allocations first
2. **Rollover Tokens** - Use from previous month's rollover
3. **Purchased Tokens** - Use from purchased packages (FIFO order)

### 3. Monthly Allocation Process
```bash
# Allocate monthly tokens for all users
php artisan tokens:allocate-monthly

# Roll over unused tokens to next month
php artisan tokens:rollover
```

## API Endpoints

### Token Usage Logs
```http
GET /api/token-usage-logs
GET /api/token-usage-logs?feature_type=sms_tokens&start_date=2024-01-01
GET /api/token-usage-logs/summary
GET /api/token-usage-logs/monthly-breakdown
GET /api/token-usage-logs/cost-analysis
```

### Usage Examples
```http
POST /api/usage/use
Content-Type: application/json
Authorization: Bearer {token}

{
    "feature_name": "transcribe_tokens",
    "amount": 15,
    "usage_context": "transcribe_audio",
    "reference_type": "audio_file",
    "reference_id": 123,
    "metadata": {
        "file_name": "podcast_episode_1.mp3",
        "duration_minutes": 15
    }
}
```

## Services

### MonthlyTokenAllocationService
Handles monthly token allocations and rollovers.

**Key Methods**:
- `allocateMonthlyTokens(User $user, string $month = null)`
- `allocateMonthlyTokensForAllUsers(string $month = null)`
- `rolloverUnusedTokens(User $user, string $fromMonth, string $toMonth)`
- `getMonthlyAllocationSummary(User $user, string $month = null)`

### Enhanced UsageService
Updated to support detailed logging and new token types.

**Key Methods**:
- `useTokens(User $user, string $featureType, int $amount, string $usageContext = null, ...)`
- `useTokensFromPurchasesWithLogging(User $user, string $featureType, int $amount, ...)`

## Commands

### Allocate Monthly Tokens
```bash
# Allocate for current month
php artisan tokens:allocate-monthly

# Allocate for specific month
php artisan tokens:allocate-monthly --month=2024-01
```

### Rollover Tokens
```bash
# Rollover from previous month to current month
php artisan tokens:rollover

# Rollover between specific months
php artisan tokens:rollover --from-month=2024-01 --to-month=2024-02
```

## Scheduled Tasks

Add to `app/Console/Kernel.php`:

```php
protected function schedule(Schedule $schedule)
{
    // Allocate monthly tokens on the 1st of each month
    $schedule->command('tokens:allocate-monthly')
        ->monthlyOn(1, '00:00')
        ->withoutOverlapping();

// Rollover unused tokens on the 1st of each month
    $schedule->command('tokens:rollover')
        ->monthlyOn(1, '01:00')
        ->withoutOverlapping();
}
```

## Migration Steps

### 1. Run Migrations
```bash
php artisan migrate
```

### 2. Seed Token Packages
```bash
php artisan db:seed --class=TokenPackageSeeder
```

### 3. Seed Plans with Token Features
```bash
php artisan db:seed --class=PlanSeeder
```

### 4. Allocate Tokens for Existing Users
```bash
php artisan tokens:allocate-monthly
```

## Usage Examples

### SMS Sending
```php
// Check if user can send SMS
if ($usageService->canUseTokens($user, 'sms_tokens', 1)) {
    // Send SMS
    $smsResult = $smsService->send($recipient, $message);
    
    if ($smsResult) {
        // Log token usage
        $usageService->useTokens(
            user: $user,
            featureType: 'sms_tokens',
            amount: 1,
            usageContext: 'sms_send',
            referenceType: 'sms',
            referenceId: $smsResult->id,
            metadata: ['recipient' => $recipient, 'message_length' => strlen($message)]
        );
    }
}
```

### Audio Transcription
```php
// Check if user can transcribe audio
$durationMinutes = $audioService->getDuration($audioFile) / 60;
if ($usageService->canUseTokens($user, 'transcribe_tokens', $durationMinutes)) {
    // Transcribe audio
    $transcriptionResult = $transcriptionService->transcribe($audioFile);
    
    if ($transcriptionResult) {
        // Log token usage
        $usageService->useTokens(
            user: $user,
            featureType: 'transcribe_tokens',
            amount: $durationMinutes,
            usageContext: 'transcribe_audio',
            referenceType: 'audio_file',
            referenceId: $audioFile->id,
            metadata: [
                'file_name' => $audioFile->name,
                'duration_minutes' => $durationMinutes,
                'transcription_id' => $transcriptionResult->id
            ]
        );
    }
}
```

## Reporting and Analytics

### Usage Summary
```php
// Get usage summary for current month
$summary = $usageService->getUserUsageSummary($user);

// Get monthly allocation summary
$allocationService = app(MonthlyTokenAllocationService::class);
$monthlySummary = $allocationService->getMonthlyAllocationSummary($user);
```

### Cost Analysis
```http
GET /api/token-usage-logs/cost-analysis?start_date=2024-01-01&end_date=2024-01-31
```

Response:
```json
{
    "success": true,
    "cost_analysis": {
        "sms_tokens": {
            "total_used": 150,
            "total_cost": 7.50,
            "avg_cost_per_token": 0.0500,
            "usage_count": 25
        },
        "transcribe_tokens": {
            "total_used": 120,
            "total_cost": 24.00,
            "avg_cost_per_token": 0.2000,
            "usage_count": 8
        }
    }
}
```

## Best Practices

### 1. Always Log Usage
- Use the enhanced `useTokens()` method with context
- Include relevant metadata for better tracking
- Reference related records when possible

### 2. Check Availability First
- Always check `canUseTokens()` before performing actions
- Provide clear error messages when limits are exceeded

### 3. Handle Monthly Allocations
- Run monthly allocation commands on schedule
- Monitor allocation success/failure rates
- Handle edge cases for plan changes

### 4. Monitor Usage Patterns
- Use the reporting endpoints to track usage trends
- Identify high-usage users for potential upgrades
- Monitor cost per token for optimization

### 5. Error Handling
- Handle cases where token allocation fails
- Provide fallback mechanisms for critical features
- Log errors for debugging and monitoring

## Troubleshooting

### Common Issues

1. **Monthly allocations not created**
   - Check if user has active subscription
   - Verify plan has token features defined
   - Check command execution logs

2. **Token usage not logged**
   - Ensure using enhanced `useTokens()` method
   - Check database connection and permissions
   - Verify token purchase relationships

3. **Rollover not working**
   - Check if rollover command is scheduled
   - Verify date formats (YYYY-MM)
   - Check for existing allocations in target month

### Debug Commands
```bash
# Check user's current allocations
php artisan tinker
>>> $user = User::find(1);
>>> $user->tokenPurchases()->monthlyAllocations()->get();

# Check usage logs
>>> $user->tokenUsageLogs()->latest()->take(10)->get();
```

TOKEN SYSTEM REDESIGN

TOKEN SYSTEM REDESIGN

📝 Page Notes

Add New Note