Getting Started with data-conductor¶

This guide will help you set up data-conductor and create your first data pipeline in just a few minutes.

Prerequisites¶

Before you begin, make sure you have:

• [ ] Access to a data-conductor instance
• [ ] Admin permissions for your organization
• [ ] Database connection details (if using SQL data steps)

Step 1: Organization Setup¶

When you first log into data-conductor, you'll need to configure your organization settings.

Setting Up Your Organization¶

1. Navigate to Organization in the main menu
2. Fill in your organization details:
3. Organization name
4. Description
5. Timezone (used for CRON scheduling)

Creating API Tokens¶

API tokens are required for certain features like API deployments.

1. Go to the API Tokens tab
2. Click Create New Token
3. Provide a name and description
4. Save the token securely (you won't see it again)

Step 2: Database Integration Setup¶

If you plan to use SQL data steps, you'll need to set up database integrations.

Adding SQL Integrations¶

1. Navigate to Integrations (this may be in your admin panel)
2. Click Add SQL Integration
3. Configure your database connection:
4. Name: A friendly name for your database
5. Type: PostgreSQL, MySQL, etc.
6. Host: Database server address
7. Port: Database port
8. Database: Database name
9. Credentials: Username and password

Testing Your Connection¶

Always test your database connection:

1. Click Test Connection
2. Verify the connection succeeds
3. Save your integration

Step 3: IP Security (Optional but Recommended)¶

For enhanced security, configure trusted IP addresses.

Adding Trusted IPs¶

1. Go to Organization → Trusted IPs tab
2. Click Add Trusted IP
3. Enter IP addresses or CIDR ranges
4. Add descriptions for easy management

Step 4: Your First Data Step¶

Now let's create a simple data step to test everything works.

Creating a SQL Data Step¶

1. Navigate to Data Tools
2. Click + New → SQL Data Step
3. Fill in the basic information:
4. Name: "Test Data Step"
5. Description: "My first data step"
6. Select your database integration
7. Write a simple SQL query:

SELECT
    'Hello data-conductor!' as message,
    NOW() as timestamp

Testing Your Query¶

1. Click the Execute button
2. Review the results in the Results tab
3. Verify your data looks correct

Saving Your Data Step¶

1. Click Save when you're satisfied
2. Your data step is now ready for deployment

Step 5: Your First Deployment¶

Let's deploy your data step so it can run automatically.

Creating a CRON Deployment¶

1. Navigate to Deployment Manager
2. Click + New Deployment → Attach Pipeline to CRON
3. Configure your deployment:
4. Name: "Daily Test Run"
5. Description: "Runs my test query daily"
6. Workflow: Select your test data step
7. Version: Use latest version
8. Environment: PROD
9. Schedule: 0 9 * * * (daily at 9 AM)

Monitoring Your Deployment¶

1. Save your deployment
2. Check the Monitoring tab to monitor runs
3. View execution logs and results

Next Steps¶

Congratulations! You've successfully set up data-conductor and created your first pipeline. Here's what to explore next:

Immediate Next Steps¶

• Create more complex data steps with variables and transformations
• Set up API endpoints to trigger pipelines on-demand
• Explore pipeline variables for dynamic queries

Advanced Features¶

• IP Security - Secure your instance
• API Integrations - Connect external systems
• Troubleshooting - Common issues and solutions

Common Issues¶

Connection Problems¶

If you're having trouble connecting to your database:

1. Check your firewall settings
2. Verify credentials are correct
3. Ensure the database server is accessible from data-conductor
4. Test with a simple tool like telnet first

Permission Issues¶

If you can't access certain features:

1. Verify your user role and permissions
2. Check with your organization administrator
3. Ensure your IP address is in the trusted list

Execution Failures¶

If your pipelines aren't running:

1. Check the execution logs in Monitoring
2. Verify your SQL syntax
3. Ensure your database integration is still working
4. Check for timeout issues with long-running queries

Need Help?¶

• FAQ - Common questions and answers
• Troubleshooting - Detailed problem-solving guide
• Contact Support - Reach out to your administrator

Ready to build more complex pipelines? Continue to Your First Pipeline!