example deploy fails Error: Invalid S3 bucket name (value: data-pipeline-simple-None-deployment-bucket)

[petervandenabeele]

The None has a capital letter which is invalid.

I ran:

export AWS_DEFAULT_ACCOUNT=_____________29
export AWS_PROFILE=my-profile
export AWS_DEFAULT_REGION=your-region # e.g. eu-west-1

<..>/datajob/examples/data_pipeline_simple$ datajob deploy --config datajob_stack.py 
cdk command: cdk deploy --app  "python <..>/datajob/examples/data_pipeline_simple/datajob_stack.py"  -c stage=None
jsii.errors.JavaScriptError: 
  Error: Invalid S3 bucket name (value: data-pipeline-simple-None-deployment-bucket)
  Bucket name must only contain lowercase characters and the symbols, period (.) and dash (-) (offset: 21)

[vincentclaes]

The None has a capital letter which is invalid.

I ran:

export AWS_DEFAULT_ACCOUNT=_____________29
export AWS_PROFILE=my-profile
export AWS_DEFAULT_REGION=your-region # e.g. eu-west-1

<..>/datajob/examples/data_pipeline_simple$ datajob deploy --config datajob_stack.py 
cdk command: cdk deploy --app  "python <..>/datajob/examples/data_pipeline_simple/datajob_stack.py"  -c stage=None
jsii.errors.JavaScriptError: 
  Error: Invalid S3 bucket name (value: data-pipeline-simple-None-deployment-bucket)
  Bucket name must only contain lowercase characters and the symbols, period (.) and dash (-) (offset: 21)

thanks Peter, I just noticed it today that this crashes with my latest changes. if you explicitly pass the stage argument it should work.

datajob deploy --config datajob_stack.py --stage dev

i plan to fix this tomorrow evening.

[petervandenabeele]

CloudFormation was successfully deployed. Now trying to run it ...

[vincentclaes]

fyi:

datajob execute --state-machine <your state machine name

to trigger the pipeline should work now

[petervandenabeele]

As mentioned in the README, I clicked "Start Execution" in the GUI and this has run successfully :-)

In the logs of the 3 AWS Glue (1.0) Jobs, I do see the "hello world" message, nice

[petervandenabeele]

The "destroy" (of the CloudFormation stack and IAM Role) also worked successfully, including the destruction of the CustomCDKBucketDeployment....

Only the CloudWatch log groups are still present. That might be OK.

Not sure if a finite Retention of 3 months or so would be better ?

/aws/lambda/data-pipeline-simple-dev-CustomCDKBucketDeployment-5N8SZ8C1NN9 | Never expire | - | - | -
-- | -- | -- | -- | --
  | /aws/lambda/data-pipeline-simple-dev-deployment-bucketBackend | Never expire | - | - | -
  | /aws/lambda/data-pipeline-simple-devBackend | Never expire | - | - | -

I will stop typing in this Issue.

I suggest to close this issue after either the README is fixed with the --stage dev explanation or some stage (e.g. dev) is set as default.

[vincentclaes]

good point, thanks Peter. we should create the cloudwatch logs as part of the stack

[petervandenabeele]

Before you do that (create snd destroy CloudWatch logs as part of CF (CloudFormation) stack): what is the life time of this CF stack? Would you want to see the Glue Job logs longer than the lifetime of this CF stack?