Migrating to Kiteworks Private Content Network

Introduction

Kiteworks offers a great software stack keeping your shared data completely fenced, secured and monitored. It offers additional features ownCloud does not provide. This guide describes how to migrate an ownCloud instance to a Kiteworks Private Content Network (PCN).

See the image below to get an overview of the building blocks of both sides.


Migrate to Kiteworks Overview

Overview

The migration works with an app to be installed on the ownCloud side. The app is provided by ownCloud as part of the guided migration. Please contact ownCloud support to get it. Both sides need to be fully configured and running, have a defined minimum release installed and are reachable from each other. If release requirements are not satisfied, you MUST upgrade first.

The migration process consists of these steps:

  • Prepare the environment.

  • Verify upcoming migration steps.

  • Run the migration.

For the planning, the ownCloud instance can stay productive during the migration process. Migration can be interrupted and restarted. For the final step, before shutting down the ownCloud instance, it is recommended to put it into maintenance mode to have a clean final transfer.

Migrated Items

The following items will be migrated. In some cases, special rules apply as noted:

  • Ordinary users.
    At Kiteworks, the email address will be used as the user name.

  • The content of the users home directory.
    The data of the home directory can be encrypted. In that case, data will be gathered, decrypted and transferred via a secured channel to the Kiteworks instance.

  • User shares received.
    As long as they reference content from another user’s home.

  • User shares granted.
    As long as they reference content from the user’s home.

  • Share permissions.
    Will be translated into Kiteworks predefined roles.

  • Guest users.
    Will be translated into Kiteworks user profile restricted.

Items Excluded from the Migration

The following items are NOT migrated. These items need to be migrated manually:

  • External mounts like
    WND, SMB, Google Drive, etc.
    This includes admin mounts and user created mounts. See the ownCloud prerequisites section for a special note.

  • S3 primary object store (S3 for the users home) cannot be migrated for the time being.

    If you have configured S3 primary as your storage location for the users home, get in touch with ownCloud support.
  • Federated shares.
    Federated shares are, in terms of data migration, like special external mounts, see above.

  • Public links.

Limitations

The following limitations impact the migration process:

  • While ownCloud fully respects letter casing for file and folder names, Kiteworks does not distinguish casings. If case conflicts happen during the migration process, a migration log file describing rclone responses and casing conflicts for files or directories is created. The ownCloud admin must resolve the conflicts to finalize the migration. For details see the Migration description.

  • When group shares have been defined, groups will not get created in Kiteworks. Instead, each member of the group will get an individual user share to the object shared.

  • Users have been able to login to ownCloud using either their display name, login name or email address. Kiteworks only allows login using the email address. The presence of the users email address in ownCloud is therefore a mandatory requirement.

  • Expiry dates for shares are not supported.

Conceptual Differences

There are some conceptual differences between the products. See the list below for important ones affecting the migration where the difference to ownCloud, if not otherwise stated, is highlighted. This list will help to identify topics addressing files, folders and shares after the migration. Note that this section does not cover using the Kiteworks instance.

  • Kiteworks cannot have files in the top level of a user’s home, only folders.
    The migration process will therefore copy all files from the users ownCloud home directory into a folder named ownCloud on the Kiteworks users top level data structure.

  • Kiteworks handles expiry dates for shares created differently. During a migration, expiry dates for ownCloud shares are ignored.

  • Shares on the ownCloud side that have been rejected by the share receiver are still potential active shares as they can be accepted at any time. This means that these shares are also migrated and the receiving share user will see them on the Kiteworks side.

  • In Kiteworks, received shares are shown at:

    • Individually shared files: in the Shared with me sidebar, not in the main file view.

    • Folders: in the main files view (outside of the ownCloud folder tree), but not in the Shared with me sidebar.

  • The filesystem on the Kiteworks side is case-insensitive.

    • Filename conflicts can happen during migration, and a migration log (csv file) will list issues that must be solved by the user.

  • Kiteworks has the following files and folder naming rules:

    • File and folder names cannot contain one of the following characters: *:"/\|<>.

    • Folder names can’t begin or end with a period.

    These rules are ineffective during the migration and this helps to complete it. But it may result in syncing issues to Windows clients. Affected files and folders can be renamed by the user. Naming rules will then be automatically enforced.

Prerequisites

To be prepared for the migration, both sides need to match the prerequisites. Please read this section carefully.

ownCloud

  • As a major prerequisite, the ownCloud instance must be running on release 10.14 or higher. If this requirement is not met, migration cannot be started as the necessary app checks the minimum version.

  • Shell/SSH access to your server running ownCloud is required.
    occ commands need to be issued.

  • The ownCloud-provided migrate-to-kiteworks app must be installed at the ownCloud instance:

  • All users must have an email address and they must be unique.
    The occ migrate:to-kiteworks:verify step will point out missing email addresses. These must be rectified before any migration can start.

  • We recommend installing and enabling, if not already present and enabled, the Impersonate app. This app can be used to solve file and folder case conflicts that can be reported during the migration process.

Installing Required Components

In all examples using the occ command we assume, that ownCloud is installed at /var/www/owncloud. Adapt the path according to your environment.

  • You must install and enable the migration app.

    • First, copy the app into the ownClouds apps or apps-external folder, preferably the latter, if it exists.

    • Set the correct user and group permissions according to your environment.

    • Finally, enable it with the following command:

      sudo -u www-data \
        php /var/www/owncloud/occ \
        app:enable migrate_to_kiteworks

External Mount Points

External mount points are not part of the automatic migration. See the following notes for a manual migration:

  • To migrate any external mount, the Kiteworks Enterprise Connect license is required.

  • If an external mount is encrypted, it must be decrypted first.

  • Follow the Kiteworks instructions to (re)connect an external mount.

  • Federated shares need, by their nature, individual treatment, no general advice can be given.

For ease of migrating external mounts, the admin should:

  • For admin created mounts, make a list of mounts with their settings and their sharing configuration.

  • For user created external mounts, the administrator is responsible to instruct users how to migrate, including how to re-setup sharing.

Kiteworks

  • As a major prerequisite, the Kiteworks instance must be running on the "Venice" release or higher. If this requirement is not met, migration cannot be started.

  • You need to login into the Kiteworks appliance as role System Admin.

  • The Kiteworks system must provide sufficient disk space for the data to be migrated. The ownCloud occ migrate:to-kiteworks:verify step will report the estimated disk space needed.

  • Ensure sufficient quota settings in the Kiteworks user profiles.

  • If it is planned to integrate Kiteworks into LDAP:

    We recommend having the Kiteworks PCN connected and configured to an LDAP server before starting the migration. This will avoid conflicting user entries that will exist in the local database additionally to the LDAP server connected.
  • If it is planned to use a virus scanner in Kiteworks:

    We recommend having the Kiteworks PCN configured using a virus scanner before starting the migration. This way, infected files will be put under quarantine already during migration.
  • In the Kiteworks Admin Console, navigate to Application Setup  Client and Plugins  API. Then click Create Custom Application:

    Kiteworks create a new custom application

    For the settings, use the following:

    • Use a speaking name

    • Check Authorization Code

    • The Access Token Lifetime can be set to the default value.

    • Set the Redirect URI to the default example value as shown when clicking into the field.
      Note, the redirect URI is not used, entering the default example is therefore ok.

  • On the next page, API Scopes are all grayed out by default.

    • Enable CREATE, READ, UPDATE, DELETE for the entities Files and Folders.

    • Now click the Add Application

  • You will get a:

    • Client application ID

    • Secret key
      Note that you only see the secret once, remember it!

    These two values are needed to initialize the ownCloud migration app.

Finally, you have the following Kiteworks values that are needed for the next steps. We recommend having them saved as shell variables for ease of use. In the upcoming examples, the following names represent the corresponding values:

  • Host name or IP address
    KW_HOST

  • Admin users eMail address
    KW_ADMIN_USER

  • Client application ID
    KW_APPLICATION_ID

  • Secret key
    KW_SECRET

Migration Steps

After the above prerequisites have been met, the migration process can be started. The process has the following steps:

  • Initialization

  • Verification

  • Migration

Both the verification and migration command need the initialisation step upfront to properly communicate with the Kiteworks instance.

Migration Initialization

The migration initialization is a mandatory step and will create a json file that is used to create a so-called "Satellite" - a trusted partner - on the Kiteworks instance.

sudo -u www-data php \
  /var/www/owncloud/occ \
  migrate:to-kiteworks:init \
  $KW_HOST \
  $KW_APPLICATION_ID \
  "$KW_SECRET"

As output, a file named mft-owncloud-migration.json is created in the ownCloud root folder. Upload this file when adding a new satellite on the Kiteworks instance. The satellite must be switched to STATUS ON.


Navigate to System Setup  Satellite Servers Add a new Satellite
Kiteworks Satellites
Kiteworks add new Satellite

Migration Verification

A potential migration must be verified upfront with a positive ready message as response. This command will also output the required space capacity needed on the Kiteworks side. The verify command currently cannot report problematic file or folder names. These are reported only during the migration process. Note that any issue reported must be solved and a verification needs to be redone before the final migration can start.

sudo -u www-data \
  php /var/www/owncloud/occ \
  migrate:to-kiteworks:verify \
  $KW_ADMIN_USER

Here are some possible verification output examples:

Example 1 - ready to migrate
Activating the Kiteworks satellite ....
Verifying users ...

Total disk storage: 13.4 MB

Congratulations - this instance is ready to be migrated to Kiteworks!
Example 2 - failure
Activating the Kiteworks satellite ....
Verifying users ...
No Email for user alex - it cannot be migrated to Kiteworks!
Please make sure all users meet the requirements.
This instance is NOT ready to be migrated to Kiteworks!

Migration

After all prerequisites, installations, configurations and the verification has passed, you can initiate the migration process. This process may take some time to copy all files to the new system. Note that a migration run can be interrupted harmlessly at any time. Starting a new migration run will continue where the previous one stopped. For a possible improvement of transfer performance, read the Tuning Transfer Performance section below. Issue the following command to start the migration:

sudo -u www-data \
  php /var/www/owncloud/occ \
  migrate:to-kiteworks \
  $KW_ADMIN_USER

The command does not require user interaction. It can be run e.g. as a screen session so that reported issues can be seen directly or as a background job. During the migration process, a log file named migrate-kiteworks-<timestamp>.csv is created in the ownCloud root folder. This file contains:

  • General rclone responses and errors,

  • rclone responses for user migration,

  • File name case conflicts that an ownCloud admin must solve.

If rclone errors at one point, it tries to finish running transfers but will stop afterwards.

Example for migration issues reported
Issues did arise when migrating files and folders.
Please review migrate-kiteworks-1712241364.csv and fix any issues which have been reported.

Once resolved please re-run the migration process again.

Migration will stop here now until no more conflicts exist.
Examples for case conflicts noted in the migration log file:
NOTICE,user1,user1@example.com,"2024/04/03 15:20:27
  NOTICE: Photos: Duplicate directory found in source - ignoring"

NOTICE,user2,user2@example.com,"2024/04/03 15:20:32
  NOTICE: Documents/Example.odt: Duplicate object found in source - ignoring"

As you can see above, there is Duplicate notice for a file and another one for a directory name. Duplicate notices are logged for case conflicts. A conflict takes place because a file or directory that has been migrated earlier is in conflict with the name of the reported object. The conflicts for the particular users need to be resolved within ownCloud. When this is done, the migration can be restarted. rclone will compare both sides to identify already migrated objects and will continue with those objects that have not been migrated yet.

Solving Case Conflicts

If there are case conflicts reported in the shell and/or the migration log, the ownCloud admin must solve them to continue the migration.

For reported conflicts, the admin should impersonate the user with the conflict and solve it by renaming the file or directory according to the Kiteworks naming rules. After fixing all open issues, the migration can be restarted and all formerly conflicted files or folders will get migrated.

Tuning Transfer Performance

By default, rclone transfers 4 files in parallel. This creates little load on the target system, but may take a longer time to complete. This is especially true when anticipating mostly small files with sizes of about 10KB instead of large files with sizes of 10 MB or above.

Performance can be tuned with the environment variable RCLONE_TRANSFERS which defines the number of concurrent file uploads.

The following example command is using 32 parallel transfers:

sudo -u www-data \
  RCLONE_TRANSFERS=32 \
  php /var/www/owncloud/occ \
  migrate:to-kiteworks \
  $KW_ADMIN_USER

Such a setting can greatly speed up the transfer of many small files, but can also lead to substantial load on the network and the target system. As a Kiteworks System Admin, it is recommended to monitor the System  Status  Performance Details pages:


Data IO System Utilization CPU System Utilization
Kiteworks Performance Details DataIO
Kiteworks Performance Details CPU

The graphs show results from a test system.

  • The left half of the graphs show the default setting with 4 parallel transfers.

  • The right half of the graphs first show RCLONE_TRANSFERS=10, then close to the end using RCLONE_TRANSFERS=32 with peaking CPU usage at near 100%.

  • During the last section, as shown in the graphs, 100 files (total of 8 MB) were uploaded per minute. The default setting would achieve only about 20 files per minute.