You want to make live changes to Daft, and test these changes on a Ray cluster.

Prerequisites:

Steps:

  1. Make changes to Daft.

  2. Build daft in release: make build-release *Make sure the .so file is < 100mb.

  3. Create a Python script to run your experiment / test.

  4. Put the script in an empty directory.

  5. SymLink the directory to the Daft directory :

    1. Example linux (from the working dir): ln -s ../daft daft .
  6. Create a runtime_env.yaml file

    1. Example:
    pip:
      packages:
        - pyarrow
        - numpy
        - tqdm
        - fsspec
      pip_check: false
    env_vars:
      PYTHONPATH: .
    
  7. Submit the job to the ray cluster:

    1. Example: ray job submit --address “[<http://localhost:8265>](<http://127.0.0.1:8265/>)” --runtime-env=”runtime_env.yaml” --working-dir “working_dir” -- python3 test.py