linux

Bitbucket Pipeline问题记录

Bitbucket Pipelines configuration reference

无法在 stage中使用 parallel

比如下面会报错

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
- stage: &deploy
        name: Deploy
        deployment: default
        parallel:
          steps:
            - step:
                name: Deploy store backend
                oidc: true
                caches:
                  - node
                script:
                  - npm i
                  - export CDK_DEPLOY_REGION=$AWS_REGION
                  - export ENVIRONMENT=$BITBUCKET_DEPLOYMENT_ENVIRONMENT
                  - export AWS_WEB_IDENTITY_TOKEN_FILE=$(pwd)/web-identity-token
                  - echo $BITBUCKET_STEP_OIDC_TOKEN > $(pwd)/web-identity-token
                  - npx cdk deploy test-$BITBUCKET_DEPLOYMENT_ENVIRONMENT-store-stack --require-approval never
            - step:
                name: Deploy store step function
                oidc: true
                caches:
                  - node
                script:
                  - npm i
                  - export CDK_DEPLOY_REGION=$AWS_REGION
                  - export ENVIRONMENT=$BITBUCKET_DEPLOYMENT_ENVIRONMENT
                  - export AWS_WEB_IDENTITY_TOKEN_FILE=$(pwd)/web-identity-token
                  - echo $BITBUCKET_STEP_OIDC_TOKEN > $(pwd)/web-identity-token
                  - npx cdk deploy test-$BITBUCKET_DEPLOYMENT_ENVIRONMENT-store-sfn-solana-stack --require-approval never

关键点在于 deployment 属性只能属于 step 或者 stage ，但是parallel 会有多个step ，但是 deployment 属性只能出现一次

Limitations

- The maximum number of steps you can within a stage is:
    - 10 steps for workspaces on the [Free plan](https://www.atlassian.com/software/bitbucket/pricing).
    - 100 steps for workspaces on a [Standard or Premium plan](https://www.atlassian.com/software/bitbucket/pricing).
- Parallel stages are not supported.
- A stage can't include parallel steps.
- A stage can't contain manually triggered steps, but you can configure a stage to be manually triggered.
- A stage can't contain conditional steps, but you can configure a stage to be conditional.
- Disabling artifact downloads inside a stage is not supported.
- Steps can't override any property also set on a stage.
- Partially completed deployment stages can't be continued if another change was subsequently deployed to the same environment.

传递文件变量

Bitbucket pipeline 可以通过Repository variables来传递变量，但是如果变量包含一些特殊字符比如换行符，bitbucket 就不能很好的处理，对于这种情况我们可以将变量用 base64 编码一下，在 pipeline 中再解码就可以解决这问题了。

1
2
3
4
cat file.txt | base64

// in pipeline file
echo ${YOUR_ENV} | base64 -d > file.txt

Condition 用法

Documentation

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
- step: &deployStepFunction
    name: Deploy store step function
    oidc: true
    caches:
      - node
    condition:
      changesets:
        includePaths:
          - "lib/sfn.ts"
          - "sfn/**"
          - "bin/**"
    script:
      - npm i
      - export CDK_DEPLOY_REGION=$AWS_REGION
      - export ENVIRONMENT=$BITBUCKET_DEPLOYMENT_ENVIRONMENT
      - export AWS_WEB_IDENTITY_TOKEN_FILE=$(pwd)/web-identity-token
      - echo $BITBUCKET_STEP_OIDC_TOKEN > $(pwd)/web-identity-token
      - npx cdk deploy test-$BITBUCKET_DEPLOYMENT_ENVIRONMENT-store-sfn-solana-stack --require-approval never

限制

  • A step within the stage can’t contain a condition. The condition must be defined on the stage.

例如，下面这段就会引发上述错误。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
image: node:18

pipelines:
  definitions:
    - stage: &deploy
        name: Deploy to Default
        deployment: default
        steps:
          - step:
              name: Install
              caches:
                - node
              script:
                - npm install
          - step:
              oidc: true
              name: Deploy Admin Stack
              condition:
                changesets:
                  includePaths:
                    - "lib/admin.ts"
              script:
                - export CDK_DEPLOY_REGION=$AWS_REGION
                - export ENVIRONMENT=$BITBUCKET_DEPLOYMENT_ENVIRONMENT
                - export AWS_WEB_IDENTITY_TOKEN_FILE=$(pwd)/web-identity-token
                - echo $BITBUCKET_STEP_OIDC_TOKEN > $(pwd)/web-identity-token
                - npx cdk deploy "project-${ENVIRONMENT}-admin-stack" --require-approval never
          - step:
              oidc: true
              name: Deploy Backend Stack
              condition:
                changesets:
                  includePaths:
                    - "lib/backend.ts"
              script:
                - export CDK_DEPLOY_REGION=$AWS_REGION
                - export ENVIRONMENT=$BITBUCKET_DEPLOYMENT_ENVIRONMENT
                - export AWS_WEB_IDENTITY_TOKEN_FILE=$(pwd)/web-identity-token
                - echo $BITBUCKET_STEP_OIDC_TOKEN > $(pwd)/web-identity-token
                - npx cdk deploy "project-${ENVIRONMENT}-backend-stack" --require-approval never

  branches:
    master:
      - stage:
          <<: *deploy
          name: Deploy to demo
          deployment: demo
    local:
      - stage:
          <<: *deploy
          name: Deploy to charles
          deployment: charles

Parallel

内存问题

pip 和 step 共享内存

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
default:
   - step:
       services:
         - redis
         - mysql
         - docker
       script:
         - echo "This step is only allowed to consume 2048 MB of memory"
         - echo "Services are consuming the rest. docker 512 MB, redis 512 MB, mysql 1024 MB"
definitions:
  services:
    redis:
      image: redis:3.2
      memory: 512
    docker:
      memory: 512  # reduce memory for docker-in-docker from 1GB to 512MB
    mysql:
      image: mysql:5.7
      # memory: 1024  # default value
      variables:
        MYSQL_DATABASE: my-db
        MYSQL_ROOT_PASSWORD: $password

OIDC Connect

Artifact

不支持环境变量名 , 比如:

1
2
3
4
5
6
7
8
9
10
script:
  - apt-get update && apt-get install -y zip
  - python3 -m pip install --upgrade pip
  - pip install poetry
  - poetry export --only $FUNCTION -f requirements.txt -o ./$FUNCTION/requirements.txt --without-hashes --without-urls
  - cd $FUNCTION
  - zip -r $FUNCTION.zip .

artifacts:
  - $FUNCTION/$FUNCTION.zip

通配符用法

1
2
artifacts:
  - '*.zip'

Multiple Lines Script

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
pipelines:
  definitions:
    - step: &deploy
        oidc: true
        caches:
          - node
        name: CDK Deploy
        script:
          - npm install
          - export AWS_ROLE_ARN=$AWS_ROLE_ARN
          - export AWS_WEB_IDENTITY_TOKEN_FILE=$(pwd)/web-identity-token
          - echo $BITBUCKET_STEP_OIDC_TOKEN > $(pwd)/web-identity-token
          - >
            if [ -z "$STACK_NAME" ]; then
              echo "STACK_NAME is not set";
              npx cdk deploy --all --require-approval never
            else
              echo "STACK_NAME is set to '$STACK_NAME'";
              npx cdk deploy $STACK_NAME --require-approval never
            fi
1
2
3
4
5
6
- >
  functions=("1" "2" "3")
  //这里要换行  不然报错
  for function in "${functions[@]}"; do
    gcloud functions deploy $(echo $function | tr '_' '-') --source=gs://$BUCKET_NAME/$function.zip --region=us-west2
  done

echo 报错问题

比如下面包含 :空格 会报错：

1
- echo "Groups to package: ${groups[@]}"

改成

1
- echo "Groups to package:\ ${groups[@]}"