Troubleshooting Guide¶

This guide helps you diagnose and resolve common issues in data-conductor. Find solutions to frequent problems, debugging tips, and when to contact support.

Quick Diagnosis¶

System Health Check¶

Before diving into specific issues, check these basics:

✅ System Status Checklist: - [ ] Can you log into data-conductor? - [ ] Are your database integrations showing as connected? - [ ] Can you access the organization settings? - [ ] Are there any error messages in the browser console?

Browser Console¶

Many issues can be diagnosed using browser developer tools:

Access Browser Console: - Chrome/Edge: F12 → Console tab - Firefox: F12 → Console tab - Safari: Cmd+Opt+C

Common Console Errors:

401 Unauthorized: Token or session expired
403 Forbidden: Permission denied
404 Not Found: Resource doesn't exist
500 Internal Server Error: Server-side issue

Authentication Issues¶

Cannot Log In¶

Symptoms: - Login page redirects back to login - "Invalid credentials" errors - Account locked messages

Solutions:

1. Check Credentials

Verify:
- Email address is correct
- Password is typed correctly
- Caps Lock is not enabled
- Browser is not auto-filling wrong credentials

2. Clear Browser Data

Clear:
- Cookies for the data-conductor domain
- Local storage
- Session storage
- Browser cache

3. Check IP Restrictions

Verify:
- Your IP is in the trusted IP list
- VPN/proxy isn't changing your IP
- Network firewall allows access
- Organization hasn't restricted access

Session Expires Quickly¶

Symptoms: - Frequent logout prompts - "Session expired" messages - Need to re-login repeatedly

Solutions:

1. Check Session Settings

Review:
- Organization session timeout settings
- Browser security settings
- Third-party cookie blocking
- Browser extensions interfering

2. Network Issues

Check:
- Stable internet connection
- Corporate proxy settings
- Firewall rules
- DNS resolution

Database Connection Issues¶

Integration Connection Failed¶

Symptoms: - "Connection failed" errors - Timeout messages during testing - Unable to execute queries

Diagnosis Steps:

1. Network Connectivity

# Test basic connectivity (if you have command line access)
telnet your-database-host 5432  # PostgreSQL
telnet your-database-host 3306  # MySQL
ping your-database-host

2. Credentials Verification

-- Test with a simple SQL client first
SELECT 1 as test;

3. Firewall and Security

Check:
- Database server firewall rules
- Cloud provider security groups
- VPC/network configuration
- SSL/TLS requirements

Slow Query Performance¶

Symptoms: - Queries taking longer than expected - Timeout errors - System becoming unresponsive

Optimization Steps:

1. Query Analysis

-- Add EXPLAIN to understand query execution
EXPLAIN (ANALYZE, BUFFERS)
SELECT customer_id, COUNT(*)
FROM orders
WHERE order_date >= '{{START_DATE}}'
GROUP BY customer_id;

2. Index Optimization

-- Create indexes on frequently filtered columns
CREATE INDEX idx_orders_date ON orders(order_date);
CREATE INDEX idx_orders_customer ON orders(customer_id);
CREATE INDEX idx_orders_date_customer ON orders(order_date, customer_id);

3. Query Optimization

-- Before: Inefficient query
SELECT * FROM large_table WHERE DATE(created_at) = '2024-01-01';

-- After: Index-friendly query
SELECT * FROM large_table
WHERE created_at >= '2024-01-01'
  AND created_at < '2024-01-02';

Pipeline Execution Issues¶

Data Step Fails to Execute¶

Symptoms: - Error messages in Results tab - "Execution failed" notifications - No results returned

Common Causes and Solutions:

1. SQL Syntax Errors

-- Check for common syntax issues:
- Missing quotes around string variables
- Incorrect variable names
- Database-specific SQL syntax
- Reserved keywords used as column names

2. Variable Issues

Verify:
- Variable names match exactly (case-sensitive)
- Test values are provided
- Variable types are correct
- No special characters causing issues

3. Permission Errors

-- Database user needs proper permissions:
GRANT SELECT ON schema.table_name TO data-conductor_user;
GRANT EXECUTE ON schema.function_name TO data-conductor_user;

Variables Not Substituting¶

Symptoms: - SQL contains literal {{VARIABLE}} text - "Column doesn't exist" errors - Unexpected query results

Solutions:

1. Variable Configuration

Check:
- Variable name spelling
- Variable is defined in Variables panel
- Test value is provided
- Variable type is appropriate

2. SQL Syntax

-- Correct variable usage:
WHERE region = '{{REGION}}'        -- String variable
WHERE amount > {{MIN_AMOUNT}}      -- Number variable
WHERE active = {{IS_ACTIVE}}       -- Boolean variable

-- Incorrect usage:
WHERE region = {{REGION}}          -- Missing quotes for string
WHERE amount > '{{MIN_AMOUNT}}'    -- Quotes around number

3. Preview Verification

Steps:
1. Switch to Preview tab
2. Verify variables are replaced
3. Check SQL syntax is correct
4. Test with simple values first

Deployment Issues¶

CRON Deployment Not Running¶

Symptoms: - No executions appearing in Jobs tab - "Next execution" time keeps updating but nothing runs - Deployment shows as active but never executes

Diagnosis Steps:

1. CRON Expression Validation

# Test your CRON expression:
# Use online tools like crontab.guru
0 9 * * *        # Daily at 9 AM - VALID
0 9 * * MON-FRI  # Weekdays at 9 AM - VALID
0 25 * * *       # Invalid hour (25) - INVALID

2. Timezone Issues

Check:
- Organization timezone setting
- Server timezone vs. expected timezone
- Daylight saving time effects
- CRON expression interpretation

3. System Resource Issues

Monitor:
- System CPU and memory usage
- Database connection pool status
- Concurrent execution limits
- Queue backlog

API Endpoint Not Responding¶

Symptoms: - 404 Not Found errors - Authentication failures - Timeout responses

Solutions:

1. Endpoint Verification

# Test API endpoint manually
curl -X POST \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"variable1": "value1"}' \
  https://your-instance.com/api/execute/deployment-id

2. Token Issues

Verify:
- Token is active and not expired
- Token has required permissions
- Authorization header format is correct
- Token is for the correct organization

3. Payload Format

// Correct payload format:
{
  "variables": {
    "START_DATE": "2024-01-01",
    "REGION": "US-West",
    "LIMIT": 100
  }
}

Performance Issues¶

Slow Page Loading¶

Symptoms: - Long delays loading pages - Partial page content loading - Browser becomes unresponsive

Solutions:

1. Browser Optimization

Actions:
- Clear browser cache and cookies
- Disable unnecessary browser extensions
- Update browser to latest version
- Try incognito/private mode

2. Network Optimization

Check:
- Internet connection speed
- Corporate proxy/firewall settings
- VPN performance impact
- DNS resolution speed

3. Data Volume

Optimize:
- Add LIMIT clauses to large queries
- Use pagination for result sets
- Filter data more aggressively
- Consider pre-aggregated views

Queries Competing for Database Resources¶

Symptoms: - Queries that used to finish quickly now time out - Several pipelines slow down at the same time each day - Your database reports connection or lock contention

This is almost always about your database rather than Data Conductor — several pipelines hitting the same tables in the same window.

Find what's running (PostgreSQL):

SELECT query, query_start, state, wait_event
FROM pg_stat_activity
WHERE state = 'active'
  AND query_start < NOW() - INTERVAL '5 minutes';

What usually fixes it: - Stagger your schedules. Deployments set to 0 * * * * all fire on the hour. Spreading them across the hour removes most contention. - Set the overlap mode on long deployments so a slow run can't overlap with its own next trigger. Choose Skip it if a late run is pointless, or Queue it if the run's work still has to happen — see Overlapping runs. - Narrow the query. Incremental processing on a date or id column beats a full scan every run. - Check the target, not the tool. Missing indexes on filter columns are the most common cause of a query that degrades as data grows.

Workflow Issues¶

A Step Is Skipped Every Run¶

A step with a condition on its incoming edge only runs when that condition passes. Open the workflow and check the edge.

The most common cause is Value from previous step data with no data to read. That condition needs the upstream data step to have Return data to pipeline enabled — the toggle under Advanced ⚙ in the SQL editor. Without it nothing is stored, and the condition reports exactly that in the run detail.

Other causes: - The field name doesn't match a column in the upstream result. Matching ignores case and underscores, but not spelling. - The upstream step returned no rows at all, so there's no first row to read. - A Previous step status condition expecting completed where the upstream step was skipped rather than run.

Open the run in Monitoring → Runs to see the reason recorded for the skip.

A Step Fails With "returned more than 5,000 rows"¶

Steps that store their results are capped. Return a summary rather than an extract — COUNT, SUM, MAX — or turn off Return data to pipeline if nothing downstream needs the values.

A Step Fails With "above the per-step limit"¶

The stored result exceeded your plan's per-step data allowance. Same fix: aggregate, or reduce the columns returned. See Monitoring for current usage.

File Ingestion Issues¶

Upload Is Blocked Before It Starts¶

The dialog needs at least one database connection to exist — there'd be nowhere to put the data. Add one under Credentials.

"Unsupported file type"¶

Accepted: CSV, TSV, Parquet, JSON, NDJSON, optionally .gz or .zst compressed. Two specific rejections:

• .zip — an archive can hold many files, so there's no single table to read. Gzip a single file instead (sales.csv.gz).
• .parquet.gz — Parquet is already compressed internally. Upload it uncompressed.

The File Loaded But Rows Are Missing¶

Check the row cap. Ingestion is built for reference data, not bulk loads — files above the row limit are rejected rather than truncated, so a partial load usually means the file itself was short, or an append mapped fewer columns than expected. Unmapped columns are skipped, never guessed.

Data Quality Issues¶

Unexpected Results¶

Symptoms: - Wrong data in query results - Missing expected rows - Incorrect calculations

Debugging Steps:

1. Data Validation

-- Check source data quality
SELECT
    COUNT(*) as total_rows,
    COUNT(DISTINCT customer_id) as unique_customers,
    MIN(order_date) as earliest_date,
    MAX(order_date) as latest_date,
    COUNT(CASE WHEN total_amount IS NULL THEN 1 END) as null_amounts
FROM orders
WHERE order_date >= '{{START_DATE}}';

2. Filter Verification

-- Test filters step by step
SELECT COUNT(*) FROM orders;  -- Total count
SELECT COUNT(*) FROM orders WHERE order_date >= '{{START_DATE}}';  -- Date filter
SELECT COUNT(*) FROM orders WHERE order_date >= '{{START_DATE}}' AND region = '{{REGION}}';  -- Combined filters

3. Join Issues

-- Check for data loss in joins
SELECT
    o.customer_id,
    COUNT(o.*) as order_count,
    CASE WHEN c.customer_id IS NULL THEN 'Missing Customer' ELSE 'Found' END as customer_status
FROM orders o
LEFT JOIN customers c ON o.customer_id = c.customer_id
GROUP BY o.customer_id, c.customer_id;

Security and Access Issues¶

Permission Denied Errors¶

Symptoms: - "Access denied" messages - 403 Forbidden errors - Unable to access certain features

Resolution Steps:

1. User Permissions

Check:
- User role (Admin, User, Viewer)
- Organization membership
- Feature-specific permissions
- Recent permission changes

2. IP Restrictions

Verify:
- Current IP is in trusted list
- CIDR ranges are correct
- VPN/proxy IP changes
- Recent IP list modifications

3. Resource Ownership

Confirm:
- User owns the resource
- Resource is shared appropriately
- Organization-level permissions
- Workspace access rights

IP Access Issues¶

Symptoms: - Blocked access from trusted locations - "IP not authorized" messages - Inconsistent access patterns

Solutions:

1. IP Detection

# Check your current IP
curl ifconfig.me
# Or visit whatismyipaddress.com

2. Network Configuration

Check:
- NAT/proxy changing your IP
- Dynamic IP assignment
- VPN endpoint changes
- Corporate firewall rules

3. CIDR Range Issues

Examples:
192.168.1.100/32  - Single IP
192.168.1.0/24    - Range 192.168.1.1-254
10.0.0.0/8        - Large corporate range

Error Code Reference¶

Common HTTP Status Codes¶

Database Error Codes¶

PostgreSQL:

42601: Syntax error
42703: Column doesn't exist
42P01: Relation doesn't exist
53300: Too many connections

MySQL:

1064: SQL syntax error
1146: Table doesn't exist
1054: Unknown column
1040: Too many connections

Getting Help¶

Self-Service Resources¶

Before Contacting Support: 1. Check this troubleshooting guide 2. Review browser console for errors 3. Test with minimal examples 4. Check system status page 5. Review recent configuration changes

Information to Gather¶

When Reporting Issues: - Exact error messages (copy/paste, not screenshots) - Steps to reproduce the problem - Browser and version you're using - Timestamp when the issue occurred - Your IP address (if relevant) - Recent changes to configuration

Contact Support¶

Support Channels: - Internal Support: Contact your organization administrator - Technical Issues: Include browser console logs - Security Issues: Report immediately with details - Feature Requests: Provide use case and business justification

Emergency Procedures¶

For Critical Issues: 1. Document the impact and affected users 2. Take screenshots of error messages 3. Note exact timestamps of failures 4. Check if issue is organization-wide or user-specific 5. Contact support immediately with all details

Prevention Tips¶

Proactive Monitoring¶

Set Up Alerts For: - Pipeline execution failures - Performance degradation - Security events - Resource usage thresholds

Regular Maintenance¶

Monthly Tasks: - Review audit logs for anomalies - Update and rotate API tokens - Clean up unused data steps and deployments - Performance optimization review

Quarterly Tasks: - Security access review - Disaster recovery testing - Documentation updates - User training refresh

Need more specific help? Check our FAQ or contact your administrator!