DXL FileTransferService config

  • Good day everyone.

    In the process of learning your technologies, i encountered a problem with using a dxlclient.config file with FileTransferService code examples.
    When trying to run basic_store_example.py, i'm catching "dxlclient.exceptions.MalformedBrokerUriException: Invalid port", what made me a bit confused , because i used that configuration file in all previous examples (event example, service example, event subscriber, event publisher, service invoker, service provider, service wrapper etc). What can be the reason of that exception?

    Also tell me if you need some additional information (screenshots, code parts etc)

  • Hi-


    I've managed that issue by editing dxlclient.config, that was located in "sample" folder (but don't know, why script used that config instead of the one i've indicated). So for now, i can't repeat that traceback, script working fine.

    But now i have another issue. When running service_provider_example.py (script 1) and then running basic_service_example.py (script 2) to send a file, script 1 tracking that file, but after completed transfer script 2 crushing with next traceback:



    UPD:

    In some iterations code working fine, in some - throwing exception, without changing any parameters.
    Another issue appeared with Basic Send File Request Example and Basic Store Example - they both running fine, but file did not transfer from one folder to another. Documentation says, that files must be saved in directory, that configured as storageDir in service configuration file, but i don't see any direction to this config file in code (for both Basic Send File Request and Basic Store).



    Thanks in advance,

    Andrew

  • Hi-


    Unfortunately, it is a bit difficult to debug what is occurring. However, I went through the process to install and test the service and it seems to be working as expected. The steps are listed below. Please let me know if you have any issues when performing these steps.


    1.) Download the latest service release


    wget https://github.com/opendxl-community/opendxl-file-transfer-service-python/releases/download/0.1.1/dxlfiletransferservice-python-dist-0.1.1.zip


    2.) Unzip the service release


    unzip dxlfiletransferservice-python-dist-0.1.1.zip


    3.) Change into service directory


    cd dxlfiletransferservice-python-dist-0.1.1


    4.) Provision the service


    dxlclient provisionconfig config <provision-service-host> service-cn


    5.) Provision the samples


    dxlclient provisionconfig sample <provision-service-host> sample-cn


    6.) Edit the service config and set a destination directory for transferred files


    vi config/dxlfiletransferservice.config


    Code: File Transfer Service Configuration
    1. ###############################################################################
    2. ## File Transfer DXL Python service settings
    3. ###############################################################################
    4. [General]
    5. # Directory under which to store files (required, no default)
    6. storageDir=/tmp
    7. ...


    7.) Start the service (force into background)


    python -m dxlfiletransferservice config &


    8.) Run the sample (send a file, "README.html")


    python sample/basic/basic_store_example.py README.html


    Code: Output
    1. 2019-06-03 16:20:42,820 dxlfiletransferclient.store INFO Assigning file id '70b7b036-79ec-4fff-98a7-17b4f8c135f6' for '/tmp/.workdir/70b7b036-79ec-4fff-98a7-17b4f8c135f6'
    2. 2019-06-03 16:20:42,821 dxlfiletransferclient.store INFO Stored file '/tmp/README.html' for id '70b7b036-79ec-4fff-98a7-17b4f8c135f6'
    3. Percent complete: 100%
    4. Response to the request for the last segment:
    5. {
    6. "file_id": "70b7b036-79ec-4fff-98a7-17b4f8c135f6",
    7. "result": "store",
    8. "segments_received": 1
    9. }
    10. Elapsed time (ms): 5.20992279053


    9.) Confirm the file was transferred


    ls -l /tmp


    Code: Output
    1. total 304
    2. -rw-r--r-- 1 root root 1209 Jun 3 16:20 README.html
    3. ...


    I transferred multiple files without experiencing any issues. Please let me know if you experience any problems when attempting to perform the above steps.


    Thanks a lot,

    Chris

  • Hi-


    I already have latest version of dxlfiletransferservice, so firstly i've started from step #3. All commands worked fine, i even had the same console output as yours, but folder is empty. Then i deleted dxlfiletransferservice folder and started from step #1, but in the end got the same result - no exceptions, no mistakes, but empty destination folder.


    And also, each time when i run dxlfiletransfer config (step #7) it appears in my broker services list. First one of this configs i'm starting from Kitematic as docker container.
    i'm working on ESXi vmware, with Linux Ubuntu (x64) operating system. Maybe, this few details will help you to understand the root of the issue.


    I also tested basic_service_example.py - it's working and sending file to indicated directory. And also, basic_send_file_request.py working fine after correct provisioning, file was transfered correctly.


    Thanks a lot,

    Andrew

  • Hi Andrew-


    The reason it is appearing multiple times in the services list, is that the service is registering each time it is started (with a unique service identifier). That occurs if the previous service was not shut down correctly. The previous instances remain until they are either invoked (by sending a request), or their TTL expires.


    If you happen to send a request to a service that no longer exists, you will receive a "ServiceNotFound" exception. The broker will round-robin service invocations. So, if you just send requests, you should see all of the stale instances disappear.


    So, what is your current status? Are you still experiencing issues?


    Thanks,

    Chris

  • Hi-


    Yes, issues are still existing. Both basic_store_example and basic_send_file_request_example scripts running normally without any exceptions or errors, they both send logging in console, that says about completed transmission, but file did not transmitted from one folder to another. Don't have any idea even where to look in code for solution


    Thanks in advance,

    Andrew

  • Hi Andrew-


    So, I would suggest following my steps exactly. I am wondering if the files themselves are being stored in the Docker container itself. In my steps, I am not using Docker at all to eliminate that possible scenario.


    Thanks,

    Chris

  • Hi Chris-


    When i made a second attempt, i've followed your instruction step by step, but nothing changed.

    Files stored on my virtual machine (but not in docker container). Directories, between which i transfer files, both placed in Documents folder.

    UPD: Looks like i've troubleshooted all that moments, for now. Issue was in absence of dxlfiletransfer.config files in some folders. But when i read the documentation,I got the impression that actions with the configuration files should be performed once and then i just need to indicate it's location. Thank you for help. Also, i would recommend to add some of your steps to documentation, so it would be easier for less-skilled users to deal with it. For example, Command Line Provisioning pages has no information about provisionconfig command for the sample folder etc.


    Thanks for help,

    Andrew

  • Great to hear you got it working!


    As far as documentation, I agree it could definitely be clearer.


    Here is the page that covers provisioning the samples for the file transfer service:


    https://opendxl-community.gith…n/pydoc/sampleconfig.html


    Each of the samples also mentions provisioning as a prerequisite:


    https://opendxl-community.gith…oc/basicstoreexample.html

    https://opendxl-community.gith…/basicserviceexample.html


    How do you think this could be improved to be clearer? Any input would be greatly appreciated!


    Thanks,

    Chris