Monorepo Support and/or filtering based on VCS changes in a specific directory

Currently, any merges to master trigger all steps of all workflows. In an monorepo that houses multiple projects, this results in multiple redundant steps as, for example, the backend is re-tested and re-deployed even if only frontend changes have been made.

 

It is currently possible to do:

 

workflows:
flow:
jobs:
- deploy:
filters:
branches:
only:
- master

 

It would be useful for all teams that run a monorepo to also be able to specify, say:

 

workflows:
flow:
jobs:
- terraform:
filters:
modified_paths:
only:
- /^infrastructure/
- deploy:
requires: terraform
filters:
modified_paths:
only:
- /^app/
  • Jamie Talbot
  • Feb 4 2018
  • Candidate
  • Attach files
  • Guest commented
    4 Aug 06:37pm

    Any updates please?

  • maor cycode commented
    31 Jul 09:28am

    Any update?

  • Hyo Jang commented
    11 Jul 01:50pm

    Any updates?

  • Danny Nemer commented
    30 Jun 06:15am

    Any updates?

  • Tim Kidd commented
    24 May 02:41am

    This is the best solution I can find. Any update on this?

    https://github.com/labs42io/circleci-monorepo

  • Piers MacDonald commented
    24 Apr 06:19am

    This is THE most requested feature and there hasn't been any progress in a year.

  • Rob Grant commented
    18 Apr 06:24pm
    We are definitely aware our monorepo support can be improved in various ways. A short-term fix will be to better surface information about the delta each trigger represents in terms of git SHAs. Longer-term, our plans are to allow more dynamic creation of config through inspection of the state of code. The idea is to give you power to do arbitrary work to then decide what the shape of your workflows should be. That work is currently sitting behind some other refactoring going on and finishing our roll out of pipelines for all projects.

    It's been a year since this. What's the update?

  • Roopak Venkatakrishnan commented
    14 Apr 03:00am
  • Phil Nielsen commented
    1 Apr 02:57pm

    I will most likely be moving at least some of our projects off of circleci and to github actions if this is not implemented or at least a timeline shared in the next few months. Definitely needed.

  • Guest commented
    30 Mar 07:28pm

    This is a very useful feature for event-based micro-service architectures where a mono-repo is used to house directories of code based on business logic domain.

  • Leo Yu commented
    28 Jan 08:11pm

    Y'all should try out Buildkite in the meantime, which does support dynamic pipeline generation.

  • Roopak Venkatakrishnan commented
    14 Oct, 2019 05:00pm

    I needed this and so I came up with an orb that has "run_if_modified" you can run only folders that are modified. Not ideal, but the best we can do now?

    https://circleci.com/orbs/registry/orb/roopakv/swissknife

  • Dumitru Deveatii commented
    9 Oct, 2019 07:02pm

    There is an article that describes how to setup monorepo with CircleCI conditional workflows and API v2. A working example can be found in this github repository.

  • Guest commented
    20 Sep, 2019 07:02pm

    Monorepos support is a really needed feature, especially given the lack of ability to reuse code across multiple repositories (as orbs are at the time of writing still a public only thing which is not good enough for most companies). Moving multiple repos into a monorepo means more code reuse of similar workflows for similar projects is possible, provided circleci can support it.

  • joe fleming commented
    27 Aug, 2019 04:14pm

    Heck yeah, that would be extremely helpful.

  • Ted Pennings commented
    13 Aug, 2019 02:25am

    I'd also like this!

  • Rob Grant commented
    22 Jul, 2019 10:53pm
    Something like what you are proposing is under design -- we are working out a few details like what to do when receiving an API call directly to a branch rather than a webhook that has a commit -- the latter has a fairly clear set of changes, but the former is somewhat more ambiguous about what "changed" (should it be from previous build vs. previous commit, for instance). We do hope to work on this once we clear out our current backlog, though, but we can't yet predict timing of release.
    @Nathan Dintenfass it's been almost a year since you wrote this - any idea of timelines?
  • Anne Nelson commented
    24 Jun, 2019 08:32pm

    Unfortunately dnephin/multirepo doesn't match the use case here, we want the pipeline not to be run at all in this scenario.

    In addition, there are no examples for the orb and it seems to have a bug where it's run twice too. It also doesn't work if you need to use 'attach_at_workspace' in your command.

  • Glen Mailer commented
    24 Jun, 2019 10:48am

    Some techniques for dealing with multiple projects in the same repo can be found in this orb: https://circleci.com/orbs/registry/orb/dnephin/multirepo

  • Matthew Morrissette commented
    13 Jun, 2019 02:10pm

    Please please please. Without this it increases the complexity and time of our builds dramatically.

  • Load older comments
  • and 395 more