Tuesday, March 3, 2015

SSIS - Best Practice and Guidelines




  • Extract data in parallel; Via using sequence container in control flow or defining different source and destination within a single Data flow Task, we can extract data parallel for  non-dependent tables or files.
    • MaxConcurrentExecutables is a package level property, by default value is -1, which mean maximum number of parallel process can be equal to no of processors + 2
    • EngineThread is a DFT level property, by default value is 10, which means maximum of 10 threads can be created to executing data flow task.
  • Extract required data; pull only the required set of data from any table or file. we need to avoid the tendency to pull everything available on the source and then delete unwanted data; it eats up network bandwidth, consumes system resources (I/O and CPU), requires extra storage, and it degrades the overall performance of ETL system.
  • Try to avoid the use of Asynchronous transformation components, that basically block the data table.
  • Limited use event in event handlers, Events are very useful but excess use of events will cost extra overhead on ETL execution.
  • Indexes on destination table should be disable, that would be re-build or re-create later.
  • Highly recommended to use "fast load" "access mode" with OLEDB destination task. Via using this option, we can change and modify some other properties like keep identity, check constraints, table lock, further more we can Rows Per Batch and Maximum insert Commit Size properties, Default value is blank, which means pull all data from source and then push it into one go, that will be overhead on tempdb.
  • Avoid implicit data type casting. For example when source is flat file then ssis implicitly cast all columns as string, that will increase data buffer size, We should be explicitly convert the data types for known columns to imporve the performance.


Reference: http://www.developer.com/db/top-10-methods-to-improve-etl-performance-using-ssis.html







No comments:

Post a Comment