Indexes

This page will try to detail each configuration available into .PyFunceble.yaml along with the location of where we are looking for the configuration file.

share_logs

Type: boolean

Default value: False

Description: Activates or disables the logs sharing.

Warning

As the underlying API and infrastructure was not changed since 2017, I choosed to remove all source code related to logs sharing.

Please consider this as reserved for future usages.

verify_ssl_certificate

Type: boolean

Default value: False

Description: Activates or disables the verification of the SSL/TLS certificate when testing for URL.

Warning

If you set this index to True, you may get false positive result.

Indeed if the certificate is not registered to the CA or is simply invalid and the domain is still alive, you will always get INACTIVE as output.

max_http_retries

Type: integer

Default value: 3

Description: Sets the maximum number of retries for an HTTP request.

debug

Type: dict

Description: Configures the debug / logging.

debug[active]

Type: boolean

Default value: False

Description: Activate or deactivates the debug / logging.

debug[level]

Type: string

Default value: info

Available values: INFO, DEBUG, WARNING, ERROR, CRITICAL.

Description: Activate or deactivates the debug / logging.

cli_decoding

Type: dict

Description: Configures everything related to the decoding of the given input sources.

cli_decoding[adblock]

Type: boolean

Default value: False

Description: Activates or disables the adblock decoding.

Note

If this index is set to True, every time we read a given file, we try to extract the elements that are present.

We basically only decode the adblock format.

Note

If this index is set to False, every time we read a given file, we will consider one line as an element to test.

cli_decoding[aggressive]

Type: boolean

Default value: False

Description: Activates or disables disable some aggressive settings related to the adblock decoding.

Warning

This option is available but please keep in mind that the underlying source code is still experimental.

cli_decoding[rpz]

Type: boolean

Default value: False

Description: Activates or disables the decoding of RPZ policies from each given input files.

cli_decoding[wildcard]

Type: boolean

Default value: False

Description: Activates or disables the decoding of wildcards for each given input files.

cli_testing

Type: dict

Description: Configures everything related to the CLI testing.

cli_testing[hosts_ip]

Type: string

Default value: "0.0.0.0"

Description: Sets the IP to prefix each lines of the hosts file.

cli_testing[max_workers]

Type: integer

Default value: null

Description: Sets the number of maximal processes workers that we are allowed to allocate for the testing.

Warning

If set to null, we use the default value calculated from your machine ressources. Meaning:

CPU cores - 2

cli_testing[autocontinue]

Type: boolean

Default value: True

Description: Activates or disables the automatic continuation subsystem.

cli_testing[inactive_db]

Type: boolean

Default value: True

Description: Activates or disables the usage of a “database” to store all INVALID and INACTIVE subject for continuous retest.

cli_testing[whois_db]

Type: boolean

Default value: True

Description: Activates or disables the uage of a “database” to store the expiration date of all domains with a valid expiration date.

Warning

We do not recomend you to disable this. In fact, this is your safety against the rate limite imposed by most WHOIS servers.

cli_testing[cidr_expand]

Type: boolean

Default value: False

Description: Activates or disables the expansion of CIDR formatted addresses.

cli_testing[complements]

Type: boolean

Default value: False

Description: Activate or disables the generation and test of the complements of a given subject.

Note

A complement is for example example.org if www.example.org is given and vice-versa.

cli_testing[cooldown_time]

Type:: float

Default value: 0.0

Description: Sets the cooldown time to apply between each test.

cli_testing[db_type]

Type: string

Default value: csv

Available values: csv, mariadb, mysql, postgresql.

Description: Sets the database type (or engine) to use everytime we create a database or a storage of a potentially huge dataset.

cli_testing[file_filter]

Type: string

Default value: null

Description: A regular expression which we use to filter the subjects to (actually) test.

cli_testing[mining]

Type: boolean

Default value: True

Description: Activates or disables the mining subsystem.

cli_testing[local_network]

Type: boolean

Default value: False

Description: Activates or disables the consideration of the test(s) in or for a local or private network context.

cli_testing[preload_file]

Type: boolean

Default value: False

Description: Activates or disables the preloading of the given input files. When this is activates, we preload the given files into the auto continue subsystem dataset in order to optimize some of our processes regarding the auto continue.

Note

This option does not have any effect if the auto continue subsystem is disabled.

cli_testing[chancy_tester]

Added in version 4.1.0b4.dev.

Type: boolean

Default value: False

Description: Activates a chancy mode that unleashes the safety workflow in place.

Warning

You shouldn’t have to use this unless you feel really lucky and trust your machine.

This mode makes things look ‘fast’, but it may produce some unexpected results if N process simultaneously write the same output file.

This mode makes the graphical CLI output unparsable - either.

MAY THE FORCE BE WITH YOU!

cli_testing[ci]

Type: dict

Description: Configures everything related to the Continuous Integration.

cli_testing[ci][active]

Type: boolean

Default value: False

Description: Activates or disables the Continuous Integration mechanism.

cli_testing[ci][commit_message]

Type: string

Default value: "PyFunceble - AutoSave"

Description: Sets the commit message to apply everytime we have to apply a commit except for the really last one.

cli_testing[ci][end_commit_message]

Type: string

Default value: "PyFunceble - Results"

Description: Sets the commit message to apply at the really end.

cli_testing[ci][max_exec_minutes]

Type: integer

Default value: 15

Description: Sets the number of minutes to wait before starting to stop a CI session.

Note

As many services are setting a rate limit per IP, it’s a good idea to set this value between 1 and 15 minute(s).

cli_testing[ci][branch]

Type: string

Default value: master

Description: Sets our git working branch. This is the branch from where we are supposed to store the tests (excepts the final results).

cli_testing[ci][distribution_branch]

Type: string

Default value: master

Description: Sets our git distributions branch. This is the branch from where we are supposed to store and push the final results.

cli_testing[ci][command]

Type: string

Default value: null

Description: Sets the command to execute before each commit (except the final one).

cli_testing[ci][end_command]

Type: string

Default value: null

Description: Sets the command to execute before the final commit.

cli_testing[display_mode]

Type: dict

Description: Configures everything related to what is displayed.

cli_testing[display_mode][dots]

Type: boolean

Default value: False

Description: Activate or disables the printing of dots or other characters when we skip the test of a subjec.

cli_testing[display_mode][dots]

Type: boolean

Default value: False

Description: Activate or disables the display of dots or other characters when we skip the test of a subjec.

cli_testing[display_mode][execution_time]

Type: boolean

Default value: False

Description: Activates or disables the display of the execution time.

cli_testing[display_mode][percentage]

Type: boolean

Default value: True

Description: Activates or disables the display and generation of the percentage - file - of each status.

cli_testing[display_mode][registrar]

Type: boolean

Default value: False

Description: Activates or disables the display and generation of the (top) registrar - file.

cli_testing[display_mode][quiet]

Type: boolean

Default value: False

Description: Activates or disables the display of output to the terminal.

Warning

If the the dots mode is activate, this option will still allow them to work.

cli_testing[display_mode][less]

Type: boolean

Default value: True

Description: Activates or disables the display of the minimal information in the table we print to stdout.

cli_testing[display_mode][all]

Type: boolean

Default value: True

Description: Activates or disables the disply of the all information in the table we print to stdout.

cli_testing[display_mode][simple]

Type: boolean

Default value: False

Description: Activates or disables the simple output mode.

Note

When this mode is active, the system will only return the result in the following format: example.org ACTIVE.

cli_testing[display_mode][status]

Type: string | list

Default value: all

Available values: all, ACTIVE, INACTIVE, INVALID, VALID, SANE, MALICIOUS

Description: Sets the status that we are allowed to print to STDOUT.

Note

A list of status can be given if you want to filter multiple status at once.

cli_testing[display_mode][colour]

Type: boolean

Default value: True

Description: Activates or disables the coloration to STDOUT.

cli_testing[display_mode][max_registrar]

Type: integer

Default value: 15

Description: Sets the maximum number of registrar to display.

Warning

This value is only used when the registrar mode is activated.

Note

This value doesn’t have any effect with the generated files.

cli_testing[testing_mode]

Type: dict

Description: Configures the testing mode to apply.

Warning

Only one of those is take in consideration.

Here is the priority / checking order:

  1. syntax

  2. reputation

  3. availability

cli_testing[testing_mode][availability]

Type: boolean

Default value: True

Description: Activates or disables the availability checker.

Note

This is the default mode.

cli_testing[testing_mode][syntax]

Type: boolean

Default value: True

Description: Activates or disables the syntax checker.

cli_testing[testing_mode][reputation]

Type: boolean

Default value: True

Description: Activates or disables the reputation checker.

cli_testing[days_between]

Type: dict

Description: Configures some days related events.

cli_testing[days_between][db_clean]

Type: integer

Default value: 28

Description: Sets the numbers of days since the introduction of a subject into the inactive dataset before it gets deleted.

Warning

As of PyFunceble 4.0.0 this is not actively implemented.

cli_testing[days_between][db_retest]

Type: integer

Default value: 28

Description: Sets the numbers of days since the introduction of a subject into the inactive dataset before it gets retested.

cli_testing[sorting_mode]

Type: dict

Description: Configures the sorting mode to apply.

Warning

Only one of those is take in consideration.

Here is the priority / checking order:

  1. hierarchical

  2. standard

cli_testing[sorting_mode][hierarchical]

Type: boolean

Default value: False

Description: Activates or disables the sorting of the files content (output) in a hierarchical order.

cli_testing[sorting_mode][standard]

Type: boolean

Default value: False

Description: Activates or disables the sorting of the files content (output) in our standard order.

cli_testing[file_generation]

Type: dict

Description: Configures everything related to the file generation.

cli_testing[file_generation][no_file]

Type: boolean

Default value: False

Description: Activates or disables the generation of any non-logs file(s).

cli_testing[file_generation][no_file]

Type: boolean

Default value: False

Description: Activates or disables the generation of any non-logs file(s).

cli_testing[file_generation][hosts]

Type: boolean

Default value: True

Description: Activates or disables the generation of the hosts file(s).

cli_testing[file_generation][plain]

Type: boolean

Default value: True

Description: Activates or disables the generation of the RAW file(s). What is meant is a list with only a list of subject (one per line).

cli_testing[file_generation][analytic]

Type: boolean

Default value: True

Description: Activates or disables the generation of the analytic file(s).

cli_testing[file_generation][unified_results]

Type: boolean

Default value: False

Description: Activates or disables the generation of the unified results file instead of the splitted one.

cli_testing[file_generation][merge_output_dirs]

Type: boolean

Default value: False

Description: Activates or disables the merging of the outputs of all inputted files inside a single subdirectory as opposed to the normal behavior.

lookup

Type: dict

Description: Configures everything related to the lookup tools to use while testing a given subject.

lookup[dns]

Type: boolean

Default value: True

Description: Activates or disables the usage of the DNS lookup whether possible.

lookup[http_status_code]

Type: boolean

Default value: True

Description: Activates or disables the usage of the HTTP status code whether possible.

lookup[netinfo]

Type: boolean

Default value: True

Description: Activates or disables the usage of the network information (or network socket) whether possible.

lookup[special]

Type: boolean

Default value: True

Description: Activates or disables the usage of our SPECIAL and extra rules whether possible.

lookup[whois]

Type: boolean

Default value: True

Description: Activates or disables the usage of the WHOIS record (or better said the expiration date in it) whether possible.

lookup[reputation]

Type: boolean

Default value: True

Description: Activates or disables the usage of the reputation dataset whether possible.

lookup[timeout]

Type: integer

Default value: 5

Description: Sets the default timeout to apply to each lookup utilities everytime it is possible to define a timeout.

lookup[platform]

Type: boolean

Default value: True

Description: Activates or disables the usage of the Platform API whether possible.

dns

Type: dict

Description: Configures everything related to the DNS lookup.

dns[server]

Type: list

Default value: null

Description: Sets the DNS server(s) to work with.

Note

When a list is given the following format is expected.

dns_server:
  - dns1.example.org
  - dns2.example.org

Note

You can specify a port number to use to the DNS server if needed.

As example:

- 127.0.1.53:5353

Warning

Please be careful when you overwrite this option. If one is not correct, you can almost sure that all results are going to be flagged as INACTIVE.

dns[protocol]

Type: string

Default value: UDP

Available values: UDP, TCP, HTTPS, TLS.

Description: Sets the protocol to use for all DNS queries.

dns[follow_server_order]

Type: boolean

Default value: True

Description: Activates or disables the follow-up of the given order.

dns[trust_server]

Type: boolean

Default value: False

Description: Activates or disables the trust mode. When the trust mode is active and the first read DNS server gives us a negative response (without any error), we take it as it is.

Otherwise, when the trust mode is disabled, when the first read DNS server gives us a negative response (without any error), we still ask all other DNS servers that were given or found.

dns[delay]

Type: float

Default value: 0.0

Description: Sets the delay to apply between each DNS query.

proxy

Added in version 4.1.0b12.dev.

Type: dict

Description: Configures everything related to the proxy settings.

proxy[global]

Added in version 4.1.0b12.dev.

Type: dict

Default value: {"http": null, "https": null}

Description:

The global proxy settings to use when no rules is matching.

The proxy settings matcher works with rules, meaning that it will first try to match any of the given rules against the hostname that is being requested against. If none is matching, it will just take over proxy[global][http] and proxy[global][https].

Note

Both http and https keys are required. But, if one is missing and the other one is given, PyFunceble will take over the given one.

If you give the following - as example:

http: http://example.org:8080
https: null

The proxy settings matcher will read/use when no rules is matched:

http: http://example.org:8080
https: http://example.org:8080

proxy[rules]

Added in version 4.1.0b12.dev.

Type: list

Default value: [] (None)

Description:

Sets the list of rules the proxy settings matcher has to follow.

The proxy settings matcher will first look at the rules listed under this key.

If you give the following - as example:

proxy:
  global:
    http: http://example.de:8080
    https: http://example.de:8080
  rules:
    - http: http://example.dev:8080
      https: http://example.dev:8080
      tld:
        - com
        - org
    - http: socks5h://example.org:8080
      https: socks5h://example.org:8080
      tld:
        - onion

The proxy settings matcher will:

  • use http://example.dev:8080 for any HTTP and HTTPS requests to a subject that has the com and org Top-Level-Domain (TLD).

    Example: example.com and example.org

  • use socks5h://example.org:8080 for any HTTP and HTTPS requests to a subject that has the onion Top-Level-Domain (TLD).

    Example: example.onion

    Warning

    If you explicitly want to use this feature to be able to test .onion subjects, be sure to use the socks5h instead of socks5 protocol.

    socks5h will delegate the DNS lookup to the proxy when socks5 will delegate the DNS lookup to the locally defined DNS resolver or server.

  • use http://example.de for any HTTP and HTTPS requests to any other subject that is not matching the previous rules.

user_agent

Type: dict

Description: Configures the user agent.

user_agent[browser]

Type: string

Default value: chrome

Available values: chrome, edge, firefox, ie, opera, safari.

Description: Sets the browser to get the get the latest user agent from.

Warning

This option is not taken in consideration if user_agent[custom] is not set to null.

user_agent[platform]

Type: string

Default value: linux

Available values: linux, macosx, win10

Description: Sets the platform to get the get the latest user agent for.

Warning

This option is not taken in consideration if user_agent[custom] is not set to null.

user_agent[custom]

Type: string

Default value: null

Description: Sets the user agent to use.

Warning

Setting this index will overwrite the choices made into user_agent[platform] and user_agent[browser].

http_codes

Type: dict

Description: Configures everything related to the HTTP status code and the way PyFunceble handles them.

http_codes[self_managed]

Type: bool

Default value: False

Description: Informs PyFunceble that the status code list should not be managed automatically.

http_codes[list]

Type: dict

Description: Categorizes the HTTP status codes.

http_codes[list][up]

Type: list

Default value:

- 100
- 101
- 102
- 200
- 201
- 202
- 203
- 204
- 205
- 206
- 207
- 208
- 226

Description: List the HTTP status codes which are considered as ACTIVE.

http_codes[list][potentially_down]

Type: list

Default value:

- 400
- 402
- 404
- 409
- 410
- 412
- 414
- 415
- 416
- 451

Description: List the HTTP status code which are considered as INACTIVE or POTENTIALLY_INACTIVE.

http_codes[list][potentially_up]

Type: list

Default value:

- 000
- 300
- 301
- 302
- 303
- 304
- 305
- 307
- 308
- 403
- 405
- 406
- 407
- 408
- 411
- 413
- 417
- 418
- 421
- 422
- 423
- 424
- 426
- 428
- 429
- 431
- 500
- 501
- 502
- 503
- 504
- 505
- 506
- 507
- 508
- 510
- 511

Description: List the HTTP status code which are considered as ACTIVE or POTENTIALLY_ACTIVE.

platform

Type: dict

Description: Configures everything related to the interaction with the Platform API.

platform[push]

Type: bool

Default value: False

Description: Activates or disables the push of the test datasets to the Platform API.

Warning

This argument is useless if the PYFUNCEBLE_PLATFORM_API environment variable is not defined.

platform[preferred_status_origin]

Type: str

Default value: frequent

Available values: frequent, latest , recommended

Description: Sets the preferred status origin when fetching data from the Platform API.