How to add a datadog.yaml configuration file?

[jeffblake]

Basic question, how do I add an agent configuration file?

I'm trying to integrate Sidekiq and need to add:

# dogstatsd_mapper_cache_size: 1000  # default to 1000
dogstatsd_mapper_profiles:
  - name: sidekiq
    prefix: "sidekiq."
    mappings:
      - match: "sidekiq.sidekiq.*"
        name: "sidekiq.*"
      - match: "sidekiq.jobs.*.perform"
        name: "sidekiq.jobs.perform"
        tags:
          worker: "$1"

to the agent datadog.yaml configuration file. Looking at the run script, it doesn't appear to be possible?

[arapulido]

Hello @jeffblake! For Heroku, we need to make some modifications to the datadog.yaml file in order to make it work correctly in this platform. My suggestion for what you want to do, would be to modify (not overwrite) the datadog.yaml file in your pre-run script (https://docs.datadoghq.com/agent/basic_agent_usage/heroku/#prerun-script). You could add there a line that modifies datadog.yaml using sed. Something similar to what we do at: https://github.com/DataDog/heroku-buildpack-datadog/blob/master/extra/datadog.sh#L56

Let me know if that works for you, and we can add documentation around it.

[jeffblake]

Thanks @arapulido , the block of configuration that I need to add (shown above) is rather unwieldy to add via sed, is it possible to add a feature for merging the example conf and a user-provided conf file?
I.e. here: https://github.com/DataDog/heroku-buildpack-datadog/blob/master/extra/datadog.sh#L27
This could read in a user provided datadog.yaml in the /datadog directory of the app, merge it with the example one, and use that as a base for the rest of the script.

[GoldenMouse]

+1 for this feature. Been trying to add a list of blacklist items for process_config and using sed is a pain.

[jeffblake]

Yes this feature would be nice! For what it's worth, I finally figured out how to do this myself using cat and a heredoc

Sidekiq integration

Add this to the prerun.sh script:
note: this only works if your conf does not already have the dogstatsd_mapper_profiles key.

cat << 'EOF' >> "$DATADOG_CONF"

dogstatsd_mapper_profiles:
  - name: sidekiq
    prefix: "sidekiq."
    mappings:
      - match: 'sidekiq\.sidekiq\.(.*)'
        match_type: "regex"
        name: "sidekiq.$1"
      - match: 'sidekiq\.jobs\.(.*)\.perform'
        name: "sidekiq.jobs.perform"
        match_type: "regex"
        tags:
          worker: "$1"
      - match: 'sidekiq\.jobs\.(.*)\.(count|success|failure)'
        name: "sidekiq.jobs.worker.$2"
        match_type: "regex"
        tags:
          worker: "$1"
EOF

May be worth adding it to the docs somewhere... I'll speak for other Heroku users and say we don't like to figure things out ourselves :)

@arapulido FYI the datadog docs here https://app.datadoghq.com/account/settings#integrations/sidekiq
has this line indented incorrectly: - match: 'sidekiq\.jobs\.(.*)\.(count|success|failure)'

[arapulido]

Thanks for the feature request! I agree this could be done better, and we will work on this

[aguynamedben]

+1 my setup is picking up log/lograge.log but I'm trying to figure out how to get Datadog to pick up an additional log, i.e. log/app.log. How do I do that?

[aguynamedben]

Ah, I figured this out once upon a time. #174

In my Rails app, we have a file datadog/conf.d/ruby.yaml

logs:
  - type: file
    path: /app/log/lograge.log
    source: ruby
    sourcecategory: sourcecode

Is this different than the purpose of the datadog.yaml file?

[arapulido]

Ah, I figured this out once upon a time. #174

In my Rails app, we have a file datadog/conf.d/ruby.yaml

logs:
  - type: file
    path: /app/log/lograge.log
    source: ruby
    sourcecategory: sourcecode

Is this different than the purpose of the datadog.yaml file?

datadog.yaml is more for general agent configuration. The ruby.yaml file is to enable and configure the Ruby integration (which includes log parsing). How to enable integrations with the buildpack is documented here: https://docs.datadoghq.com/agent/basic_agent_usage/heroku/#enabling-integrations