Migrating to Kiteworks Private Content Network

Introduction

Kiteworks offers a great software stack keeping your shared data completely fenced, secured and monitored. It offers additional features ownCloud does not provide. This guide describes how to migrate an ownCloud instance to a Kiteworks Private Content Network (PCN).

See the image below to get an overview of the building blocks of both sides.


Migrate to Kiteworks Overview

Overview

The migration works with an app to be installed on the ownCloud side. The app is provided by ownCloud as part of the guided migration. Please contact ownCloud support to get it. Both sides need to be fully configured and running, have a defined minimum release installed and are reachable from each other. If release requirements are not satisfied, you MUST upgrade first.

The migration process consists of these steps:

  • Prepare the environment.

  • Verify upcoming migration steps.

  • Run the migration.

For the planning, the ownCloud instance can stay productive during the migration process. Migration can be interrupted and restarted. For the last step, which is migrating shares, you must put the instance into maintenance mode to have a clean final transfer. Afterwards you can finally shut down the ownCloud instance.

Migrated Items

The following items will be migrated. In some cases, special rules apply as noted:

  • Ordinary users.
    At Kiteworks, the email address will be used as the user name.

  • Guest users.
    Will be translated into Kiteworks user profile restricted.

  • Disabled users.
    Disabled users cant login, but may contain shares to be migrated.

  • The content of the users home directory.
    The data of the home directory can be encrypted. In that case, data will be gathered, decrypted and transferred via a secured channel to the Kiteworks instance.

  • User shares received.
    As long as they reference content from another user’s home.

  • User shares granted.
    As long as they reference content from the user’s home. A share where all permissions are already granted thru a parent folder are considered redundant and are not created.

  • Share permissions.
    Will be translated into Kiteworks predefined roles.

Items Excluded from the Migration

The following items are NOT migrated. These items need to be migrated manually:

  • External mounts like
    WND, SMB, Google Drive, etc.
    This includes admin mounts and user created mounts. See the ownCloud prerequisites section for a special note.

  • S3 primary object store (S3 for the users home) cannot be migrated for the time being.

    If you have configured S3 primary as your storage location for the users home, get in touch with ownCloud support.

  • Federated shares.
    Federated shares are, in terms of data migration, like special external mounts, see above.

  • Public links.
    They are not supported on Kiteworks and skipped during the migration process.

Limitations

The following limitations impact the migration process:

  • The ownCloud migration app is only functional on amd64 platforms.

  • While ownCloud fully respects letter casing for file and folder names, Kiteworks does not distinguish casing. If case conflicts happen during the migration process, a migration log file describing rclone responses and casing conflicts for files or directories is created. The ownCloud admin must resolve the conflicts to finalize the migration. For details see the Migration description.

  • When group shares have been defined, groups will not get created in Kiteworks. Instead, each member of the group will get an individual user share to the object shared.

  • In ownCloud, users can login either using their display name, login name or email address. Kiteworks only allows login using the email address. The presence of the users email address in ownCloud, which must be unique, is therefore a mandatory requirement.

Conceptual Differences

There are some conceptual differences between the products. See the list below for important ones affecting the migration where the difference to ownCloud, if not otherwise stated, is highlighted. This list will help to identify topics addressing files, folders and shares after the migration. Note that this section does not cover using the Kiteworks instance.

  • Kiteworks cannot have files in the top level of a user’s home, only folders.
    The migration process will therefore copy the entire hierarchy of the ownClouds user home into a folder named ownCloud on the Kiteworks users top level data structure.

  • Kiteworks handles expiry dates for shares created differently. During a migration, expiry dates for ownCloud shares are ignored.

  • Shares on the ownCloud side that have been rejected by the share receiver are still potential active shares as they can be accepted at any time. This means that these shares are also migrated and the receiving share user will see them on the Kiteworks side.

  • In Kiteworks, received shares are shown at:

    • Individually shared files:
      in the Shared with me sidebar, not in the main file view.

    • Folders:
      in the main files view (outside of the ownCloud folder tree), but not in the Shared with me sidebar.

  • The filesystem on the Kiteworks side is case-insensitive.

    • Filename conflicts can happen during migration, and a migration log will list issues that must be solved by the admin.

  • Kiteworks has the following files and folder naming rules:

    • File and folder names cannot contain the following characters: *?:"/\|<>.

    • Folder names can’t begin or end with a period.

    These rules are ineffective during the migration and this helps to complete it. But it may result in syncing issues to Windows clients. Affected files and folders can be renamed by the user. Naming rules will then be enforced.

Prerequisites

To be prepared for the migration, both sides need to match the prerequisites. Please read this section carefully.

ownCloud

  • As a major prerequisite, the ownCloud instance must be running on release 10.14 or higher. If this requirement is not met, migration cannot be started as the necessary app checks the minimum version.

  • For all migration steps, the ownCloud instance must run in normal operation mode. Migration is not possible if ownCloud is in maintenance mode.

  • Shell/SSH access to your server running ownCloud is required.
    occ commands need to be issued.

  • The ownCloud-provided migrate-to-kiteworks app must be installed at the ownCloud instance:

  • All users must have an email address and they must be unique.
    The occ migrate:to-kiteworks:verify step will point out missing email addresses. These must be rectified before any migration can start.

  • We recommend installing and enabling, if not already present and enabled, the Impersonate app. This app can be used for example to solve file and folder case conflicts that can be reported during the migration process.

Installing Required Components

In all examples using the occ command we assume, that ownCloud is installed at /var/www/owncloud. Adapt the path according to your environment.

  • You must install and enable the migration app.

    • First, copy the app into the ownClouds apps or apps-external folder, preferably the latter, if it exists.

    • Set the correct user and group permissions according to your environment.

    • Finally, enable it with the following command:

      sudo -u www-data \
  php /var/www/owncloud/occ \
  app:enable migrate_to_kiteworks

External Mount Points

External mount points are not part of the automatic migration. See the following notes for a manual migration:

  • To migrate any external mount, the Kiteworks Enterprise Connect license is required.

  • If an external mount is encrypted, it must be decrypted first.

  • Follow the Kiteworks instructions to (re)connect an external mount.

  • Federated shares need, by their nature, individual treatment, no general advice can be given.

For ease of migrating external mounts, the admin should:

  • For admin created mounts, make a list of mounts with their settings and their sharing configuration.

  • For user created external mounts, the administrator is responsible to instruct users how to migrate, including how to re-setup sharing.

Kiteworks

  • As a major prerequisite, the Kiteworks instance must be running version 8.6 or higher.

  • You need to login into the Kiteworks appliance as role System Admin.

  • The Kiteworks system must provide sufficient disk space for the data to be migrated. The ownCloud occ migrate:to-kiteworks:verify step will report the estimated disk space needed.

  • Ensure sufficient quota settings in the Kiteworks user profiles.

  • If it is planned to integrate Kiteworks into LDAP:

    We recommend having the Kiteworks PCN connected and configured to an LDAP server before starting the migration. This will avoid conflicting user entries that will exist in the local database additionally to the LDAP server connected.

  • If it is planned to use a virus scanner in Kiteworks:

    We recommend having the Kiteworks PCN configured using a virus scanner before starting the migration. This way, infected files that have not been covered by ownCloud will be put under quarantine already during migration.

  • In the Kiteworks Admin Console, navigate to Application Setup  Client and Plugins  API. Then click Create Custom Application:

    Kiteworks create a new custom application

    For the settings, use the following:

    • Use a speaking name

    • Check Authorization Code

    • The Access Token Lifetime can be set to the default value.

    • Set the Redirect URI to the default example value as shown when clicking into the field.
      Note, the redirect URI is not used, entering the default example is therefore ok.

  • On the next page, API Scopes are all grayed out by default.

    • Enable CREATE, READ, UPDATE, DELETE for the entities Files and Folders.

    • Now click the Add Application

  • You will get a:

    • Client application ID

    • Secret key
      Note that you only see the secret once, remember it!

    These two values are needed to initialize the ownCloud migration app.

Finally, you have the following Kiteworks values that are needed for the next steps. In the upcoming examples, the following names represent the corresponding values:

  • Host name or IP address
    KW_HOST

  • Admin users email address
    KW_ADMIN_USER

  • Client application ID
    KW_APPLICATION_ID

  • Secret key
    KW_SECRET

Consider saving KW_ADMIN_USER as shell variable for ease of use in the following commands.

Migration Steps

After the above prerequisites have been met, the migration process can be started. The process has the following steps:

  • Initialization

  • Verification

  • Migration

    • Migrate users

    • Migrate files

    • Migrate shares

    • Disable users

Both the verification and migration commands need the initialisation step upfront to properly communicate with the Kiteworks instance.

Details for commands used can be found in the Migrate to Kiteworks occ command description.

Migration Initialization

The migration initialization is a mandatory step and will create a json file that is used to create a so-called "Satellite" - a trusted partner - on the Kiteworks instance.

sudo -u www-data php \
  /var/www/owncloud/occ \
  migrate:to-kiteworks:init \
  KW_HOST \
  KW_APPLICATION_ID \
  "KW_SECRET"

As output, a file named mft-owncloud-migration.json is created in the ownCloud root folder. Use this file now to create a new satellite on the Kiteworks instance. The satellite must be switched to STATUS ON.


Navigate to System Setup  Satellite Servers Add a new Satellite
Kiteworks Satellites
Kiteworks add new Satellite

Migration Verification

A migration must be verified upfront with a positive ready message as response. This command will also output a rough estimate of the required space capacity needed on the Kiteworks side. The verify command currently cannot report problematic file or folder names. These are reported only during the migration process. Note that any issue reported must be solved and a verification needs to be redone before the migration can start.

sudo -u www-data \
  php /var/www/owncloud/occ \
  migrate:to-kiteworks:verify \
  $KW_ADMIN_USER

Here are some possible verification output examples:

Example 1 - ready to migrate
Activating the Kiteworks satellite ....
Verifying users ...

Total disk storage: 13.4 MB

Congratulations - this instance is ready to be migrated to Kiteworks!
Example 2 - failure
Activating the Kiteworks satellite ....
Verifying users ...
No email for user alex - it cannot be migrated to Kiteworks!
Please make sure all users meet the requirements.
This instance is NOT ready to be migrated to Kiteworks!

Migration

After all prerequisites, installations, configurations and the verification has passed, you can initiate the migration process. The migration is split into four parts which are:

1. Migrate users (1)
2. Migrate files (1)
3. Migrate shares (2)
4. Disable users (3)
1 These steps migrate all ordinary, guest and disabled users and files. You can rerun these steps to migrate any items that did not exist, or failed to migrate when the respective step was called. Note that the steps must be made in that order.
2 This step is a breaking change and migrates all shares. When this step has run, the former steps cant be run again.
3 This step finalizes the migration by disabling all users on Kiteworks that are disabled on ownCloud.

The migration step transferring files will naturally take its time depending on the amount of data and bandwidth available. All other steps will complete relative quickly as only metadata is transferred.

All migration steps, especially files, can be interrupted harmlessly at any time. Starting a new migration run will continue where the previous one stopped.

Migrate Users

Issue the following command to start migrating users:

sudo -u www-data \
  php /var/www/owncloud/occ \
  migrate:to-kiteworks:users \
  $KW_ADMIN_USER \
  KW_PROFILE_GUEST (optional)

Note that you can optionally add a Kiteworks guest user profile that will be assigned to guest users defined in ownCloud when migrating. If this is not assigned, the default restricted will be used.

The command does not require user interaction. It can be run e.g. as a screen session so that reported issues can be seen directly or as a background job.

Migrate Files

Issue the following command to start migrating files:

sudo -u www-data \
  php /var/www/owncloud/occ \
  migrate:to-kiteworks:files \
  $KW_ADMIN_USER

The command does not require user interaction. It can be run e.g. as a screen session so that reported issues can be seen directly or as a background job. During the migration process, a log file named migrate-kiteworks-files.csv is created in the ownCloud root folder. This file contains:

  • General rclone responses and errors,

  • rclone responses for user migration,

  • File name case conflicts that an ownCloud admin must solve.

If rclone errors at one point, it tries to finish running transfers but will stop afterwards.

Example for migration issues reported
Issues did arise when migrating files and folders.
Please review migrate-kiteworks-1712241364.csv and fix any issues which have been reported.

Once resolved please re-run the migration process again.

Migration will stop here now until no more conflicts exist.
Examples for case conflicts noted in the migration log file
NOTICE,user1,user1@example.com,"2024/04/03 15:20:27
  NOTICE: Photos: Duplicate directory found in source - ignoring"

NOTICE,user2,user2@example.com,"2024/04/03 15:20:32
  NOTICE: Documents/Example.odt: Duplicate object found in source - ignoring"

As you can see above, there is Duplicate notice for a file and another one for a directory name. Duplicate notices are logged for case conflicts. A conflict takes place because a file or directory that has been migrated earlier is in conflict with the name of the reported object. The conflicts for the particular users need to be resolved within ownCloud. When this is done, the migration can be restarted. rclone will compare both sides to identify already migrated objects and will continue with those objects that have not been migrated yet.

Migrate Shares

When running this command, you cant run the migrate:files|users command anymore!

Issue the following command to start migrating shares:

sudo -u www-data \
  php /var/www/owncloud/occ \
  migrate:to-kiteworks:shares \
  $KW_ADMIN_USER

The command does not require user interaction. It can be run e.g. as a screen session so that reported issues can be seen directly or as a background job. During the migration process, a log file named migrate-kiteworks-shares.csv is created in the ownCloud root folder.

Disable Users

The migrate users step migrates all users as enabled users even they were disabled ownCloud users. This is needed so that a disabled user can provide shares. As a final step, run this command to disable users at Kiteworks that are disabled at ownCloud.

Issue the following command to start disable users:

sudo -u www-data \
  php /var/www/owncloud/occ \
  migrate:to-kiteworks:disable-users \
  $KW_ADMIN_USER

The command does not require user interaction. It can be run e.g. as a screen session so that reported issues can be seen directly or as a background job.

Solving Case Conflicts

If there are case conflicts reported in the shell and/or the migration log, the ownCloud admin must solve them to continue the migration.

For reported conflicts, the admin should impersonate the user with the conflict and solve it by renaming the file or directory according to the Kiteworks naming rules. After fixing all open issues, the migration can be restarted and all formerly conflicted files or folders will get migrated.