Skip to main content

How to move multiple files in ssis and also rename simultaneously

There are two ways to achieve this.
1) We can move the flat files and then rename it.
2) While moving files itself, automatic rename should be done.

We will do the second type. The criteria is to rename the files while moving from source to destination.
So for that, we need FILE SYSTEM TASK to be included. Secondly since we need to move many files, we will use FOR EACH LOOP CONTAINER.

To fetch all the files, we can use FOR EACH LOOP task in SSIS.
In collection tab, we can select FOREACH FILE enumerator option for fetching files and we can change enumerator configuration
Folder option: Points to source where we need to fetch files.
Files: will give us idea whether we need to fetch all the files (*.*) or if we give extension like *.txt, it is going to fetch only  .txt files .
Once I give Source name in FOR EACH LOOP container, It is going to fetch all the files corresponding to that path.

Retrieve file name:
This option is used to let the variables mentioned in VARIABLE MAPPING tab, like how fetch the file and in what format.

This is supported by 3 options:
1) Fully qualified
2) Name and extension
3) Name only

1) Fully qualified: Fully qualified will assign path and file name with its extension to variable present in VARIABLE MAPPING.

2) Name and extension: If we use name and extension then, Only name and extension will be assigned to variable at VARIABLE MAPPING option

3) Name Only: Name only option enables variable at VARIABLE MAPPING option to fetch only name of file (Not even the extension)

Say example: If we have three text files in our source like txt1, txt2, txt3 and we need to fetch all three text files.

Now these files which are actually my source are in the path C:\Documents and Settings\swami\Desktop\test\source

Now I want to move these files to C:\Documents and Settings\swami\Desktop\test\dest.

My requirement is that while moving these files, I should rename these files and should place to destination folder 'Dest'.


Now at FOR EACH LOOP container, We will give source path where our files resides at Folder option from collection tab.


secondly I'm going to give *.txt in files position to retrieve only .txt files.

Retrieve file name is selected as Name and extension and only name and extension will be passed to any variable I assign to VARIABLE MAPPING tab.

Now I'm assigning a dummy variable named by me as 'Filename' to VARIABLE MAPPING tab.

Now variable Filename will take only Name of file and extension from given path.

ie,. Values of Filename taken by foreach loop container at first iteration during run will be only txt1.txt
and on second iteration, it will be txt2.txt and so on.. Iteration counts are handled by foreach loop container internally.

now for renaming, Im going to create one more variable with values being the path of source. ie., I create a variable named 'source' and value is C:\Documents and Settings\swami\Desktop\test\source.\

|||rly Im creating one more variable for destination named 'destination' and values assigned will be C:\Documents and Settings\swami\Desktop\test\dest.

Now I'm adding a FILE SYSTEM TASK inside foreach loop container.

I'm creating one more variable called 'sourcepath'.
Reason for creating this variable is  that 'souce' variable assign only source path name and filename variable have only files name with  its extension.

to integrate path and then corresponding file loop fetches with extension, I created this variable.
The expression for variable sourcepath is given as user::source+user::filepath.

ie,. on first iteration, Sourcepath will have value as C:\Documents and Settings\swami\Desktop\test\source\txt1.txt.
 and on second iteration, it will have values like  C:\Documents and Settings\swami\Desktop\test\source\txt2.txt. and so on..

I'm assigning this variable Source path to File System task in source column.

Similar things are done on destination also by creating new variable 'destinationpath' where its expression is given as user::destination+'newname_'+user::filepath

where filepath is txt1.txt, txt2.txt and so on variable destination is indicates to destination path.

After running package, The files are transferred with their name are changed. The complete snippets are shown follows:-



 


Comments

  1. Hi Dhinakaran S,

    Great article,
    and how to do the file name extend with date. In above example txt1_2012-05-10.txt,txt2_2012-05-10.txt and txt3_2012-05-10.txt.

    This is an urgent issue, Please anyone could help me on this.


    Thanks
    Anil

    ReplyDelete
    Replies
    1. How To Move Multiple Files In Ssis And Also Rename Simultaneously >>>>> Download Now

      >>>>> Download Full

      How To Move Multiple Files In Ssis And Also Rename Simultaneously >>>>> Download LINK

      >>>>> Download Now

      How To Move Multiple Files In Ssis And Also Rename Simultaneously >>>>> Download Full

      >>>>> Download LINK W1

      Delete
  2. Hello Anil.

    In Foreach loop editor, Under Retrieve Filename, instead of Name and Extension, you can use Name Only and user::filename variable will hold only the name of file.
    Now use this variable to append the date and extension at last.
    Example: if ur filename is abc.txt, then variable user::filename will hold 'abc' and then in FILESYSTEM task,
    Use this to append the date function as I explained earlier in same post with Extension ('.txt' in your case).

    Hope you are cleared.

    ReplyDelete
  3. I think I'm about to go crazy! Thanks so much for the example... I've replicated it (3 times from scratch so far)...but it won't run successfully. Here is the error I get. Any idea what I'm doing wrong? Oh... so frustrated.

    TITLE: Package Validation Error
    ------------------------------

    Package Validation Error

    ------------------------------
    ADDITIONAL INFORMATION:

    Error at File System Task: Failed to lock variable "D:\SSIS Jobs\Jobs_Prepense\Example SSIS Jobs\Move_RenameFiles\Source\" for read access with error 0xC0010001 "The variable cannot be found. This occurs when an attempt is made to retrieve a variable from the Variables collection on a container during execution of the package, and the variable is not there. The variable name may have changed or the variable is not being created.".

    Error at File System Task [File System Task]: An error occurred with the following error message: "Failed to lock variable "D:\SSIS Jobs\Jobs_Prepense\Example SSIS Jobs\Move_RenameFiles\Source\" for read access with error 0xC0010001 "The variable cannot be found. This occurs when an attempt is made to retrieve a variable from the Variables collection on a container during execution of the package, and the variable is not there. The variable name may have changed or the variable is not being created.".
    ".

    Error at File System Task: Failed to lock variable "D:\SSIS Jobs\Jobs_Prepense\Example SSIS Jobs\Move_RenameFiles\dest\_" for read access with error 0xC0010001 "The variable cannot be found. This occurs when an attempt is made to retrieve a variable from the Variables collection on a container during execution of the package, and the variable is not there. The variable name may have changed or the variable is not being created.".

    Error at File System Task [File System Task]: An error occurred with the following error message: "Failed to lock variable "D:\SSIS Jobs\Jobs_Prepense\Example SSIS Jobs\Move_RenameFiles\dest\_" for read access with error 0xC0010001 "The variable cannot be found. This occurs when an attempt is made to retrieve a variable from the Variables collection on a container during execution of the package, and the variable is not there. The variable name may have changed or the variable is not being created.".
    ".

    Error at File System Task: There were errors during task validation.

    (Microsoft.DataTransformationServices.VsIntegration)

    ------------------------------
    BUTTONS:

    OK
    ------------------------------

    ReplyDelete
    Replies
    1. Hello Jim,

      This error means, that you have not configured your variable properly in FILE SYSTEM task. Please check the issue what variable you have used.

      Delete
    2. I am also facing same error and couldn't resolve it.

      Delete
  4. Nice information. Thanks for sharing content and such nice information for me. I hope you will share some more content about. Please keep sharing!
    big data training in chennai

    ReplyDelete
  5. Your blog is very interesting. And the point describing the information very effectively.
    Big Data Institutes in Pune

    ReplyDelete
  6. Your good knowledge and kindness in playing with all the pieces were very useful. I don’t know what I would have done if I had not encountered such a step like this.
    Best PHP Training Institute in Chennai|PHP Course in chennai

    Best .Net Training Institute in Chennai
    Software Testing Training in Chennai
    Blue Prism Training in Chennai
    Angularjs Training in Chennai

    ReplyDelete
  7. I think there are actually more and more useful aspects of SSIS and has more functions in order to solve complex IT problems.

    SSIS Upsert

    ReplyDelete
  8. Thanks for the nice Post. I must suggest your readers to Visit Big data course in coimbatore

    ReplyDelete
  9. How To Move Multiple Files In Ssis And Also Rename Simultaneously >>>>> Download Now

    >>>>> Download Full

    How To Move Multiple Files In Ssis And Also Rename Simultaneously >>>>> Download LINK

    >>>>> Download Now

    How To Move Multiple Files In Ssis And Also Rename Simultaneously >>>>> Download Full

    >>>>> Download LINK X2

    ReplyDelete

Post a Comment

Popular posts from this blog

BIG Data, Hadoop – Chapter 2 - Data Life Cycle

Data Life Cycle The data life cycle is pictorial defined as show below:     As we see, in our current system, we capture/ Extract our data, then we store it and later we process for reporting and analytics. But in case of big data, the problem lies in storing and then processing it faster. Hence Hadoop takes this portion, where it stores the data in effective format (Hadoop distributed File System) and also process using its engine (Map Reduce Engine). Since Map Reduce engine or Hadoop engine need data on HDFS format to process, We have favorable tools available in market to do this operation. As an example, Scoop is a tool which converts RDBMS to HDFS. Likewise we have SAP BOD to convert sap system data to HDFS.

SSIS: The Value Was Too Large To Fit In The Output Column

I had a SSIS package where I was calling a stored procedure in OLEDB Source and it was returning a “The Value Was Too Large to Fit in the Output Column” error. Well, My Datatype in OLEDB source was matching with my OLEDB Destination table. However, when I googled, we got solutions like to increase the output of OLEDB Source using Advanced Editor option . I was not at all comfortable with their solution as my source, destination and my intermediate transformation all are having same length and data type and I don’t want to change. Then I found that I was missing SET NOCOUNT ON option was missing in Stored Procedure. Once I added it, my data flow task ran successfully. 

How to Copy or Move Multiple Files from One Folder to Another Folder using Talend

Hello all, In this Post, I will explain how to move Multiple Files from One Folder (Say Source) to Other folder (Say Destination). This Post will also helps you to understand How to Declare Variable and Use it. To Declare a variable, We are go to use Contexts option in repository. Lets say we have two .txt files in Path D:/Source/ . My Requirement is to move the files from Source Folder ( D:/Source/ ) to Destination Folder ( D:/Dest/ ). Step 1: Open a New job Step 2: Now right click and Create a New Contexts from Repository. Give some Name and give Next. Step 3: Now Fill in the Source Directory Details where the loop on files should happen as shown in the snippet and give finish. Step 4: Now Context is created and The values will be changing based on each file in Folder. Step 5: Click and Drag the context from Repository to Context Job Window below the Job Designer. Step 6: If we Expand the Contexts, We can find the variable SourcePath is holdi...