Skip to main content

How to Copy or Move Multiple Files from One Folder to Another Folder using Talend

Hello all,

In this Post, I will explain how to move Multiple Files from One Folder (Say Source) to Other folder (Say Destination).

This Post will also helps you to understand How to Declare Variable and Use it.

To Declare a variable, We are go to use Contexts option in repository.

Lets say we have two .txt files in Path D:/Source/. My Requirement is to move the files from Source Folder (D:/Source/) to Destination Folder (D:/Dest/).

Step 1: Open a New job
Step 2: Now right click and Create a New Contexts from Repository. Give some Name and give Next.




Step 3: Now Fill in the Source Directory Details where the loop on files should happen as shown in the snippet and give finish.




Step 4: Now Context is created and The values will be changing based on each file in Folder.

Step 5: Click and Drag the context from Repository to Context Job Window below the Job Designer.



Step 6: If we Expand the Contexts, We can find the variable SourcePath is holding value the Path of Source Folder and Also, Script Code is generated.

This Script code is what we are going to use in coming process.



Step 7: In Order to iterates on files or Folder, We are going to use  tFileList component. This  tFileList component is found under Files>Management from the Palette on Right Side.

tFileList component acts as a Foreach Loop Container with File Enumerator as in SSIS.
Click and Drag the  tFileList Component to designer Space Since we need to work on multiple files from Source Path D:/Source/ We will configure  tFileList component to fetch each file one by one.

Now Since we are working on only TEXT Files/Flat Files, I will configure to fetch only .txt files from the folder. Any other files other than txt files will be ignored.

This can be done by simply adding "*.txt" in FileMask option under Files Block in  tFileList component.
Don't forget to use " " Double Quotes. Its a Mandatory.


Since variable FilePath which is already holding path of the Source, We simply can give context.FilePath in Directory option in component which now tFileList understands which folder it should work.

Also FileListType is selected as FILES from the Drop Down.
To Summarize, The values to be configured in tFileList components are 

Directory: context.FilePath 
FileListType: Files
Files: "*.txt"

Step 8: Now our tFileList  is configured to fetch all the files. Now to copy the files from D:/Source/ to D:/Dest/ folder, we will drag and drop tFileCopy Component from Files>Management under palette.

tFileCopy Copies a source file into a target directory and can remove the source files if required.

Also we will first connect our tFileList component to tFileCopy component.
(Note: The connection will be iterate as shown in snippet)



Now, we lets configure tFileCopy to copy the file.

  • We know that path for the file is present in variable context.FilePath. But to get the exact file name, we should write a small command in tFileCopy.
  • This can be done by simply adding ((String)globalMap.get("tFileList_1_CURRENT_FILEPATH")) in filename of tFileCopy component.
  • Configure the Destination Directory as shown in the snippet.



And Thats it. Run the job and check the file move.


Comments

  1. Thank you. Detailed description helps. :)

    ReplyDelete
  2. Hello Dhinakaran, How do you send two different files 1) Error file 2) Successful files to two different folders -- let us say both are coming from above process -- ( one folder multiple files)

    ReplyDelete
  3. Hello Dhinakaran, thanks. How can i do to list all files copied and send their names in a mail??

    ReplyDelete
  4. Nice information. Thanks for sharing content and such nice information for me. I hope you will share some more content about. Please keep sharing!
    big data training in chennai

    ReplyDelete
  5. I feel this is a very good information and post about SSIS and many other useful aspects.On the other hand it is a very useful tool to solve complex IT problems.

    SSIS Upsert

    ReplyDelete
  6. Thanks for such informative Post. I must suggest your readers to Visit Big data course in coimbatore

    ReplyDelete
  7. Really very nice information,thank you for sharing your thoughts with us .
    thank you....

    Big data online training

    Big data hadoop training

    ReplyDelete
  8. with us to get some idea about that content.Thanks for sharing
    Msbi Online Course
    Msbi Online Training

    ReplyDelete
  9. Great article. Thank you for sharing such informative content. If you are looking for Digital marketing company in Dubai then go through it.

    ReplyDelete

Post a Comment

Popular posts from this blog

Zip/Unzip multiple files and also include password for zipped file using SSIS

We have many scenario that we need to Zip many files which we come across and then so some operations like either sending it as a email or just moving zipped file to some other destinations etc. But we were using manual method to zip multiple files. In this post, I tried to create a package which will zip multiple files using SSIS. Here for Zipping files purpose, I'm using 7-ZIP which is free software available in google sites. Download files and install onto your system. First let me show how to Zip on file and later I will show how to zip multiple files using SSIS and 7Zip tool. Compressing Single file. Here I'm trying to Zip one single flat file which is of 40MB size. I kept this file in C:\Documents and Settings\\Desktop\test\source folder. Now to compress this file, I will open my SSIS and I'm dragging and dropping EXECUTE PROCESS TASK from Control Flow. Now right click on Execute Process task and go for edit and select Process option. In process tab,

SSIS: The Value Was Too Large To Fit In The Output Column

I had a SSIS package where I was calling a stored procedure in OLEDB Source and it was returning a “The Value Was Too Large to Fit in the Output Column” error. Well, My Datatype in OLEDB source was matching with my OLEDB Destination table. However, when I googled, we got solutions like to increase the output of OLEDB Source using Advanced Editor option . I was not at all comfortable with their solution as my source, destination and my intermediate transformation all are having same length and data type and I don’t want to change. Then I found that I was missing SET NOCOUNT ON option was missing in Stored Procedure. Once I added it, my data flow task ran successfully. 

How to move multiple files in ssis and also rename simultaneously

There are two ways to achieve this. 1) We can move the flat files and then rename it. 2) While moving files itself, automatic rename should be done. We will do the second type. The criteria is to rename the files while moving from source to destination. So for that, we need FILE SYSTEM TASK to be included. Secondly since we need to move many files, we will use FOR EACH LOOP CONTAINER. To fetch all the files, we can use FOR EACH LOOP task in SSIS. In collection tab, we can select FOREACH FILE enumerator option for fetching files and we can change enumerator configuration Folder option: Points to source where we need to fetch files. Files: will give us idea whether we need to fetch all the files (*.*) or if we give extension like *.txt, it is going to fetch only  .txt files . Once I give Source name in FOR EACH LOOP container, It is going to fetch all the files corresponding to that path. Retrieve file name: This option is used to let the variables mentioned in VARIA