Data Vacuum Library

A modular data import library for .NET 8 that fetches data from SQL Server, CSV files, or REST APIs and maps it into your APS system. Supports single-row imports, mapping transformations, and a pluggable handler system for entity-specific logic.

📋 Table of Contents

1. Features
2. Architecture
3. Getting Started
4. Installation
5. Configuration
6. Usage
7. Adding New Handlers
8. Extending Connectors
9. Examples
10. Dependencies
11. Contributing

✨ Features

• Multi-source data fetching:
  • SQL Server (SqlConnector)
  • REST APIs (ApiConnector)
  • CSV files (CsvFileConnector)
• Dynamic mapping engine to convert external fields to APS fields
• Pluggable handler system via ITargetUpsertHandler for entity-specific logic
• Chained imports support for parent-child relationships
• Case-insensitive field mapping for enhanced robustness
• Easily extensible architecture for new connectors and entities

🏗️ Architecture

[ImportService]
    ├── [ConnectorFactory]
    │   ├── [SqlConnector]
    │   ├── [ApiConnector]
    │   └── [CsvFileConnector]
    ├── [MappingEngine]
    │   └── applies ImportMapping rules
    └── [ITargetUpsertHandler]
        ├── [OrderHandler]
        ├── [PostHandler]
        └── [CustomHandlers...]

Core Components

• ImportService: Orchestrates the entire import process for a single entity
• ConnectorFactory: Returns the appropriate connector based on source type
• MappingEngine: Applies ImportMapping rules and field transformations
• ITargetUpsertHandler: Implements entity-specific persistence and processing logic

🚀 Getting Started

Prerequisites

• .NET 8 SDK
• SQL Server (if using SqlConnector)
• Internet access (if using ApiConnector)

Clone Repository

git clone git@git.case-tunisia.com:Internal/data-vacuum.git
cd data-vacuum

Build Solution

dotnet build

📦 Installation

Package Manager

dotnet add package DataVacuum

NuGet Package Manager Console

Install-Package DataVacuum

⚙️ Configuration

Register the required services in your Program.cs:

using DataVacuum.Extensions;

var builder = WebApplication.CreateBuilder(args);

// Register APS Data Import services
builder.Services.AddDataVacuum(builder.Configuration);

// Register your custom handlers
builder.Services.AddScoped<ITargetUpsertHandler, OrderUpsertHandler>();
builder.Services.AddScoped<ITargetUpsertHandler, PostHandler>();

// Configure HTTP client for API connector
builder.Services.AddHttpClient<ApiConnector>();

var app = builder.Build();

Configuration Options

Add to your appsettings.json:

{
  "DataVacuum": {
    "DefaultConnectionString": "Server=localhost;Database=APS;Trusted_Connection=true;",
    "ApiTimeout": "00:05:00",
    "BatchSize": 1000
  }
}

💻 Usage

Basic Import Example

using DataVacuum.Services;
using DataVacuum.Models;

// Create service scope
using var scope = app.Services.CreateScope();
var importService = scope.ServiceProvider.GetRequiredService<ImportService>();

// Configure import
var config = new ImportConfig
{
    EntityName = "Posts",
    SourceType = DataSourceType.Api,
    ConnectionJson = @"{
        ""url"": ""https://jsonplaceholder.typicode.com/posts"",
        ""method"": ""GET"",
        ""headers"": {}
    }",
    Mappings = new List<ImportMapping>
    {
        new() { ExternalField = "userId", ApsField = "UserId" },
        new() { ExternalField = "id", ApsField = "PostId" },
        new() { ExternalField = "title", ApsField = "Title" },
        new() { ExternalField = "body", ApsField = "Content" }
    }
};

// Run the import
try
{
    await importService.RunImportAsync(config);
    Console.WriteLine("Import completed successfully!");
}
catch (Exception ex)
{
    Console.WriteLine($"Import failed: {ex.Message}");
}

SQL Server Import

var sqlConfig = new ImportConfig
{
    EntityName = "Orders",
    SourceType = DataSourceType.Sql,
    ConnectionJson = @"{
        ""connectionString"": ""Server=localhost;Database=Source;Trusted_Connection=true;"",
        ""query"": ""SELECT OrderId, CustomerId, OrderDate, TotalAmount FROM Orders WHERE OrderDate > @startDate"",
        ""parameters"": {
            ""@startDate"": ""2024-01-01""
        }
    }",
    Mappings = new List<ImportMapping>
    {
        new() { ExternalField = "OrderId", ApsField = "Id" },
        new() { ExternalField = "CustomerId", ApsField = "CustomerId" },
        new() { ExternalField = "OrderDate", ApsField = "CreatedDate" },
        new() { ExternalField = "TotalAmount", ApsField = "Total" }
    }
};

await importService.RunImportAsync(sqlConfig);

CSV File Import

var csvConfig = new ImportConfig
{
    EntityName = "Products",
    SourceType = DataSourceType.CsvFile,
    ConnectionJson = @"{
        ""filePath"": ""C:\\Data\\products.csv"",
        ""hasHeader"": true,
        ""delimiter"": "","",
        ""encoding"": ""UTF-8""
    }",
    Mappings = new List<ImportMapping>
    {
        new() { ExternalField = "product_id", ApsField = "Id" },
        new() { ExternalField = "product_name", ApsField = "Name" },
        new() { ExternalField = "price", ApsField = "Price", Transform = "decimal" },
        new() { ExternalField = "category", ApsField = "CategoryName" }
    }
};

await importService.RunImportAsync(csvConfig);

🔧 Adding New Handlers

1. Implement ITargetUpsertHandler

using DataVacuum.Interfaces;

public class ArticleHandler : ITargetUpsertHandler
{
    private readonly IArticleRepository _articleRepository;
    private readonly ILogger<ArticleHandler> _logger;

    public string EntityName => "Articles";

    public ArticleHandler(IArticleRepository articleRepository, ILogger<ArticleHandler> logger)
    {
        _articleRepository = articleRepository;
        _logger = logger;
    }

    public async Task UpsertAsync(IDictionary<string, object> mapped, CancellationToken ct = default)
    {
        try
        {
            var article = new Article
            {
                Id = Convert.ToInt32(mapped["Id"]),
                Title = mapped["Title"]?.ToString(),
                Content = mapped["Content"]?.ToString(),
                AuthorId = Convert.ToInt32(mapped["AuthorId"]),
                PublishedDate = Convert.ToDateTime(mapped["PublishedDate"])
            };

            await _articleRepository.UpsertAsync(article, ct);
            _logger.LogInformation("Successfully processed article: {ArticleId}", article.Id);
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Failed to process article with data: {@MappedData}", mapped);
            throw;
        }
    }
}

2. Register in Dependency Injection

// In Program.cs
builder.Services.AddScoped<ITargetUpsertHandler, ArticleHandler>();

🔌 Extending Connectors

Create Custom Connector

using DataVacuum.Interfaces;
using DataVacuum.Models;

public class MyCustomConnector : IDataConnector
{
    private readonly ILogger<MyCustomConnector> _logger;
    private readonly IMyCustomDataSource _dataSource;

    public MyCustomConnector(ILogger<MyCustomConnector> logger, IMyCustomDataSource dataSource)
    {
        _logger = logger;
        _dataSource = dataSource;
    }

    public async Task<IEnumerable<IDictionary<string, object>>> FetchRowsAsync(
        ImportConfig config, 
        CancellationToken ct = default)
    {
        try
        {
            // Parse connection configuration
            var connectionConfig = JsonSerializer.Deserialize<MyCustomConnectionConfig>(
                config.ConnectionJson);

            // Fetch data from your custom source
            var rawData = await _dataSource.GetDataAsync(connectionConfig, ct);

            // Convert to dictionary format
            return rawData.Select(item => new Dictionary<string, object>
            {
                ["Id"] = item.Id,
                ["Name"] = item.Name,
                ["Value"] = item.Value
                // Map other properties as needed
            });
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Failed to fetch data from custom source");
            throw;
        }
    }
}

Register Custom Connector

// In Program.cs
builder.Services.AddScoped<IDataConnector, MyCustomConnector>();

// Update ConnectorFactory to include your custom connector
builder.Services.Configure<ConnectorOptions>(options =>
{
    options.RegisterConnector(DataSourceType.Custom, typeof(MyCustomConnector));
});

📖 Examples

Advanced Mapping with Transformations

var mappings = new List<ImportMapping>
{
    new() 
    { 
        ExternalField = "price", 
        ApsField = "Price", 
        Transform = "decimal",
        DefaultValue = "0.00"
    },
    new() 
    { 
        ExternalField = "is_active", 
        ApsField = "IsActive", 
        Transform = "boolean"
    },
    new() 
    { 
        ExternalField = "created_date", 
        ApsField = "CreatedDate", 
        Transform = "datetime",
        TransformFormat = "yyyy-MM-dd HH:mm:ss"
    }
};

Chained Imports (Parent-Child)

// Import customers first
var customerConfig = new ImportConfig
{
    EntityName = "Customers",
    // ... configuration
};
await importService.RunImportAsync(customerConfig);

// Then import orders with customer references
var orderConfig = new ImportConfig
{
    EntityName = "Orders",
    ParentEntityName = "Customers",
    // ... configuration
};
await importService.RunImportAsync(orderConfig);

📚 Dependencies

• .NET 8: Target framework
• Microsoft.Extensions.DependencyInjection: Dependency injection
• Microsoft.Extensions.Logging: Logging abstraction
• System.Data.SqlClient: SQL Server connectivity
• CsvHelper: CSV file processing
• Newtonsoft.Json: JSON serialization

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.