We're extremely excited to open source Orchestra! Learn more by reading the blog post.

ORCHESTRA

An open source system to orchestrate teams of experts and machines on complex projects.
Humans where they're best, machines for the rest.

Orchestra orchestrates teams (e.g., reporters and photographers in a newsroom) and projects (e.g., a news story). In Orchestra workflows, you can assign senior experts to review other experts in order to provide feedback and iteratively refine work. Orchestra also brings automation, like classifiers or crawlers written in Python, onto projects to help out. New workflows can be added with some simple Python glue and an html interface.

Example Workflow
Below we'll walk you through an example of how Orchestra could be used in a newsroom. Take a look at the example implementation in our documentation!
  • An editor finds a good story and kicks off the reporting workflow, providing information to help a reporter get started.
  • The reporter picks up the available story and writes up a draft article.
  • A more experienced reporter then reviews the article and suggests improvements.
  • In parallel with the reporting step, a photographer picks up the available story as well, capturing relevant photos.
  • A senior photographer reviews the photos, making edits and selecting the best ones.
  • The selected photos are resized and recolored for display across different media.
  • Finally, a copy editor adds headlines and photo captions to complete the story.
Concepts
We'll now walk you through major Orchestra concepts based on the example above.

Workflows

  • The entire process above is called a workflow, comprised of five component steps.
  • Two of these steps require review, where more experienced experts review the original work performed. Custom review policies (e.g., sampled or systematic review) for tasks can be easily created in Orchestra.
  • The photo resizing step is a machine step, completed by automation rather than by experts.
  • Each step emits a JSON blob with structured data generated by either humans or machines.
  • Steps have access to data emitted by previous steps that they depend on. In the example, the copy editor has access to both the story and the resized photos.

Project Distribution

  • Projects are a series of interconnected tasks. A project is an instance of a workflow; a task is an instance of a step.

    An editor with a story about local elections would create an elections project, with tasks for a reporter/photographer/copy editor.

  • Tasks are carried out by an expert or by a machine.

    Photographers capture the story.

    Machines resize and recolor the photos.

Experts can come from anywhere, from a company's employees to freelancers on platforms like Upwork.

Hierarchical Review

  • Core experts do the initial work on a task.

  • Reviewers provide feedback to other experts to make their work even better.

  • The core expert submits the task when their work is complete.

  • The reviewer can choose to accept the task, which is either selected for further review or marked as complete.

  • They could also choose to return the task, requesting changes from and giving feedback to the worker they are reviewing.

Worker Certification

  • Certifications allow experts to work on tasks they're great at.

  • Experts can work toward all sorts of certifications, picking up practice tasks to build experience.

    Joseph is a solid reporter but needs a little more practice as a photographer—let's give him some simple tasks so he can improve!

  • Experts need additional certification to work in a reviewer role.

    Amy has been reporting for quite some time and would be great at mentoring new reporters.

 Life of a Task

Below are two images of the Orchestra dashboard, the launching point for expert workers.
Click to see how tasks move differently across the dashboard for core workers and reviewers.

Core Expert

Reviewer

Community

Orchestra is a project built around people. We're developers and designers who love to think about how we can take small steps toward improving the world and we love meeting people who feel the same! Reach out to us with requests for help, feature requests, suggestions, complaints, and compliments on our Gitter channel. Alternatively, reach out over our mailing list!

Motivation
Team
Unlimited Labs has open sourced Orchestra as part of our goal to build a brighter future of work.

We are a startup based in NYC that is passionate about improving how people do creative and analytical work. We have a strong team of engineers and designers who have worked extensively on systems that help people work productively online.

Beyond focusing on profit, we believe that the products and experiences we design should be considerate of their greater social context and impact. To stay true to these values, we are in the process of becoming a B-certified corporation.

Research
Orchestra is motivated by years of research into machine-mediated expert teams.

Flash teams from Stanford, a study in empowering managers to coordinate interactions between a team of experts contributing to a larger project. The Foundry project is a mature research prototype of these ideas.

Review hierarchies work from Anand Kulkarni et al. and Daniel Haas et al. shows that machines can pair experts with other experts to improve work quality while facilitating mentorship.

Active learning, or human-in-the-loop machine learning, is the study of how machine learning model training can happen in concert with a domain expert completing their work.