Configuration
Location
Problematics
How can we create a more efficient way to work with configuration?
How can we make the configuration file(s) available globally so that PyFunceble can be run everywhere in the user workspace?
To answer those problematics, we moved the configuration location elsewhere in the place where most users expect to have their configuration file(s).
Filename-s
At any of the listed configuration location, the following file will be read:
.PyFunceble.yamlThis file is generated automatically if missing.
This file is automatically replaced if you choose to merge the upstream configuration when a features key is introduced or removed.
This “new” behavior was introduced at the same time .PyFunceble.overwrite.yaml was. This is to ensure PyFunceble at all time is running on a fully compatible version of .PyFunceble.yaml
.PyFunceble.overwrite.yamlThis is the project specific configuration file for PyFunceble. This file is now taking presedence over
.PyFunceble.yamlwhich for sure previously was the file you used to alter for your pwn needs.This file can only be maintained by a human-controlled process. PyFunceble will never write into and never owerwrite this file, once it is created.
Repository clone
If you cloned the repository and you’re trying to test from a cloned
directory (the one with for example CONTRIBUTING.md) we consider
the configuration directory as the current one.
Note
This behavior allows us to not modify the way we develop PyFunceble.
Travis CI
Under Travis CI, we search or initiate the configuration at the directory we are currently located.
Warning
We don’t care about the distribution, as long as the
TRAVIS_BUILD_DIR environment variable is set, we search or
initiate the configuration in the current directory.
Note
If you want to force the directory where we should work, you can
initiate the PYFUNCEBLE_CONFIG_DIR environment variable with
the path where we should work.
GitLab CI/CD
Under GitLab CI/CD, we search or initiate the configuration at the directory we are currently located.
Warning
We don’t care about the distribution, as long as the PROJECT_CI
and GITLAB_CI environment variables are set, we search or
initiate the configuration in the current directory.
Note
If you want to force the directory where we should work, you can
initiate the PYFUNCEBLE_CONFIG_DIR environment variable with
the path where we should work.
GitHub Actions
Under GitHub Actions, we search or initiate the configuration at the directory we are currently located.
Warning
We don’t care about the distribution, as long as the
GITHUB_ACTIONS environment variable is set, we search or
initiate the configuration in the current directory.
Note
If you want to force the directory where we should work, you can
initiate the PYFUNCEBLE_CONFIG_DIR environment variable with
the path where we should work.
Jenkins CI
Under Jenkins CI, we search or initiate the configuration at the directory we are currently located.
Warning
We don’t care about the distribution, as long as the
JENKINS_URL and JENKINS_HOME environment variables
are set, we search or initiate the configuration in the current
directory.
Note
If you want to force the directory where we should work, you can
initiate the PYFUNCEBLE_CONFIG_DIR environment variable with
the path where we should work.
Linux and MacOS (Darwin Kernel)
Under Linux and MacOS, we look for the following directories in their order. If any configuration directory is found, the system proposes you to install them automatically on the first configuration file.
~/.config/PyFunceble~/.PyFunceble${PWD}
Note
If the parent directory does not exist, we move to the next possible location in the given order.
This means that under most Linux distributions and MacOS versions,
we consider ~/.config/PyFunceble as the configuration
location.
But if the ~/.config directory does not exist, we fallback
to ~/.PyFunceble as the configuration location.
Windows
As mentioned by Pat Altimore’s Blog, we used the
Per user configuration files synchronized across domain joined
machines via Active Directory Roaming section to understand what we
should do to find our configuration directory.
Under Windows, we look for the following directories in their order. If any configuration directory is found, the system proposes you to install them automatically on the first configuration file.
%APPDATA%PyFunceble(environnement variable)%CD%
Note
%CD% is explained by the set command (set /?):
%CD% - expands to the current directory string.
Note
If the parent directory does not exist, we move to the next possible location in the given order.
This means that under most Windows versions, we consider
%APPDATA%PyFunceble - also know as
C:UsersuserNameAppDataRoamingPyFunceble- as the
configuration location.
But if the %APPDATA% directory does not exist, we fall back
to the current directory as the configuration location.
Custom location
Sometimes, you may find yourself in a position where you absolutely do not want PyFunceble to use its default configuration location.
For that reason, if you set your desired configuration location along
with the PYFUNCEBLE_CONFIG_DIR environment variable, we take
that location as the (default) configuration location.
Autoconfiguration
Sometimes, you may find yourself in a position that you do not or you can’t answer the question which asks you if you would like to merge an upstream configuration file into your local one.
For that reason, if you can set the PYFUNCEBLE_AUTO_CONFIGURATION
as an environment variable. Setting that environment variable will make
PyFunceble merge the upstream configuration when a new key is available but
not found into your configuration file.
Warning
As of 4.0.0 the configuration file is automatically copied into
your configuration directory - if it is not found.
Indexes
This page will try to detail each configuration available into .PyFunceble.yaml along with the location of where we are looking for the configuration file.
verify_ssl_certificate
Type:
booleanDefault value:
FalseDescription: Activates or disables the verification of the SSL/TLS certificate when testing for URL.
Warning
If you set this index to True, you may get false positive
result.
Indeed if the certificate is not registered to the CA or is simply
invalid and the domain is still alive, you will always get
INACTIVE as output.
max_http_retries
Type:
integerDefault value:
3Description: Sets the maximum number of retries for an HTTP request.
debug
Type:
dictDescription: Configures the debug / logging.
debug[active]
Type:
booleanDefault value:
FalseDescription: Activate or deactivates the debug / logging.
debug[level]
Type:
stringDefault value:
infoAvailable values:
INFO,DEBUG,WARNING,ERROR,CRITICAL.Description: Activate or deactivates the debug / logging.
cli_decoding
Type:
dictDescription: Configures everything related to the decoding of the given input sources.
cli_decoding[adblock]
Type:
booleanDefault value:
FalseDescription: Activates or disables the adblock decoding.
Note
If this index is set to True, every time we read a given file, we
try to extract the elements that are present.
We basically only decode the adblock format.
Note
If this index is set to False, every time we read a given file, we
will consider one line as an element to test.
cli_decoding[aggressive]
Type:
booleanDefault value:
FalseDescription: Activates or disables disable some aggressive settings related to the adblock decoding.
Warning
This option is available but please keep in mind that the underlying source code is still experimental.
cli_decoding[rpz]
Type:
booleanDefault value:
FalseDescription: Activates or disables the decoding of RPZ policies from each given input files.
cli_decoding[wildcard]
Type:
booleanDefault value:
FalseDescription: Activates or disables the decoding of wildcards for each given input files.
cli_testing
Type:
dictDescription: Configures everything related to the CLI testing.
cli_testing[hosts_ip]
Type:
stringDefault value:
"0.0.0.0"Description: Sets the IP to prefix each lines of the hosts file.
cli_testing[max_workers]
Type:
integerDefault value:
nullDescription: Sets the number of maximal processes workers that we are allowed to allocate for the testing.
Warning
If set to null, we use the default value calculated from your
machine ressources. Meaning:
CPU cores - 2
cli_testing[autocontinue]
Type:
booleanDefault value:
TrueDescription: Activates or disables the automatic continuation subsystem.
cli_testing[inactive_db]
Type:
booleanDefault value:
TrueDescription: Activates or disables the usage of a “database” to store all
INVALIDandINACTIVEsubject for continuous retest.
cli_testing[whois_db]
Type:
booleanDefault value:
TrueDescription: Activates or disables the uage of a “database” to store the expiration date of all domains with a valid expiration date.
Warning
We do not recomend you to disable this. In fact, this is your safety against the rate limite imposed by most WHOIS servers.
cli_testing[cidr_expand]
Type:
booleanDefault value:
FalseDescription: Activates or disables the expansion of CIDR formatted addresses.
cli_testing[complements]
Type:
booleanDefault value:
FalseDescription: Activate or disables the generation and test of the complements of a given subject.
Note
A complement is for example example.org if
www.example.org is given and vice-versa.
cli_testing[cooldown_time]
Type::
floatDefault value:
0.0Description: Sets the cooldown time to apply between each test.
cli_testing[db_type]
Type:
stringDefault value:
csvAvailable values:
csv,mariadb,mysql,postgresql.Description: Sets the database type (or engine) to use everytime we create a database or a storage of a potentially huge dataset.
cli_testing[file_filter]
Type:
stringDefault value:
nullDescription: A regular expression which we use to filter the subjects to (actually) test.
cli_testing[mining]
Type:
booleanDefault value:
TrueDescription: Activates or disables the mining subsystem.
cli_testing[local_network]
Type:
booleanDefault value:
FalseDescription: Activates or disables the consideration of the test(s) in or for a local or private network context.
cli_testing[preload_file]
Type:
booleanDefault value:
FalseDescription: Activates or disables the preloading of the given input files. When this is activates, we preload the given files into the auto continue subsystem dataset in order to optimize some of our processes regarding the auto continue.
Note
This option does not have any effect if the auto continue subsystem is disabled.
cli_testing[chancy_tester]
Added in version 4.1.0b4.dev.
Type:
booleanDefault value:
FalseDescription: Activates a chancy mode that unleashes the safety workflow in place.
Warning
You shouldn’t have to use this unless you feel really lucky and trust your machine.
This mode makes things look ‘fast’, but it may produce some unexpected results if
Nprocess simultaneously write the same output file.This mode makes the graphical CLI output unparsable - either.
MAY THE FORCE BE WITH YOU!
cli_testing[ci]
Type:
dictDescription: Configures everything related to the Continuous Integration.
cli_testing[ci][active]
Type:
booleanDefault value:
FalseDescription: Activates or disables the Continuous Integration mechanism.
cli_testing[ci][commit_message]
Type:
stringDefault value:
"PyFunceble - AutoSave"Description: Sets the commit message to apply everytime we have to apply a commit except for the really last one.
cli_testing[ci][end_commit_message]
Type:
stringDefault value:
"PyFunceble - Results"Description: Sets the commit message to apply at the really end.
cli_testing[ci][max_exec_minutes]
Type:
integerDefault value:
15Description: Sets the number of minutes to wait before starting to stop a CI session.
Note
As many services are setting a rate limit per IP, it’s a good idea to set
this value between 1 and 15 minute(s).
cli_testing[ci][branch]
Type:
stringDefault value:
masterDescription: Sets our git working branch. This is the branch from where we are supposed to store the tests (excepts the final results).
cli_testing[ci][distribution_branch]
Type:
stringDefault value:
masterDescription: Sets our git distributions branch. This is the branch from where we are supposed to store and push the final results.
cli_testing[ci][command]
Type:
stringDefault value:
nullDescription: Sets the command to execute before each commit (except the final one).
cli_testing[ci][end_command]
Type:
stringDefault value:
nullDescription: Sets the command to execute before the final commit.
cli_testing[display_mode]
Type:
dictDescription: Configures everything related to what is displayed.
cli_testing[display_mode][dots]
Type:
booleanDefault value:
FalseDescription: Activate or disables the printing of dots or other characters when we skip the test of a subjec.
cli_testing[display_mode][dots]
Type:
booleanDefault value:
FalseDescription: Activate or disables the display of dots or other characters when we skip the test of a subjec.
cli_testing[display_mode][execution_time]
Type:
booleanDefault value:
FalseDescription: Activates or disables the display of the execution time.
cli_testing[display_mode][percentage]
Type:
booleanDefault value:
TrueDescription: Activates or disables the display and generation of the percentage - file - of each status.
cli_testing[display_mode][registrar]
Type:
booleanDefault value:
FalseDescription: Activates or disables the display and generation of the (top) registrar - file.
cli_testing[display_mode][quiet]
Type:
booleanDefault value:
FalseDescription: Activates or disables the display of output to the terminal.
Warning
If the the dots mode is activate, this option will still allow them to work.
cli_testing[display_mode][less]
Type:
booleanDefault value:
TrueDescription: Activates or disables the display of the minimal information in the table we print to stdout.
cli_testing[display_mode][all]
Type:
booleanDefault value:
TrueDescription: Activates or disables the disply of the all information in the table we print to stdout.
cli_testing[display_mode][simple]
Type:
booleanDefault value:
FalseDescription: Activates or disables the simple output mode.
Note
When this mode is active, the system will only return the result in the
following format: example.org ACTIVE.
cli_testing[display_mode][status]
Type:
string|listDefault value:
allAvailable values:
all,ACTIVE,INACTIVE,INVALID,VALID,SANE,MALICIOUSDescription: Sets the status that we are allowed to print to STDOUT.
Note
A list of status can be given if you want to filter multiple status at once.
cli_testing[display_mode][colour]
Type:
booleanDefault value:
TrueDescription: Activates or disables the coloration to STDOUT.
cli_testing[display_mode][max_registrar]
Type:
integerDefault value:
15Description: Sets the maximum number of registrar to display.
Warning
This value is only used when the registrar mode is activated.
Note
This value doesn’t have any effect with the generated files.
cli_testing[testing_mode]
Type:
dictDescription: Configures the testing mode to apply.
Warning
Only one of those is take in consideration.
Here is the priority / checking order:
syntaxreputationavailability
cli_testing[testing_mode][availability]
Type:
booleanDefault value:
TrueDescription: Activates or disables the availability checker.
Note
This is the default mode.
cli_testing[testing_mode][syntax]
Type:
booleanDefault value:
TrueDescription: Activates or disables the syntax checker.
cli_testing[testing_mode][reputation]
Type:
booleanDefault value:
TrueDescription: Activates or disables the reputation checker.
cli_testing[days_between]
Type:
dictDescription: Configures some days related events.
cli_testing[days_between][db_clean]
Type:
integerDefault value:
28Description: Sets the numbers of days since the introduction of a subject into the inactive dataset before it gets deleted.
Warning
As of PyFunceble 4.0.0 this is not actively implemented.
cli_testing[days_between][db_retest]
Type:
integerDefault value:
28Description: Sets the numbers of days since the introduction of a subject into the inactive dataset before it gets retested.
cli_testing[sorting_mode]
Type:
dictDescription: Configures the sorting mode to apply.
Warning
Only one of those is take in consideration.
Here is the priority / checking order:
hierarchicalstandard
cli_testing[sorting_mode][hierarchical]
Type:
booleanDefault value:
FalseDescription: Activates or disables the sorting of the files content (output) in a hierarchical order.
cli_testing[sorting_mode][standard]
Type:
booleanDefault value:
FalseDescription: Activates or disables the sorting of the files content (output) in our standard order.
cli_testing[file_generation]
Type:
dictDescription: Configures everything related to the file generation.
cli_testing[file_generation][no_file]
Type:
booleanDefault value:
FalseDescription: Activates or disables the generation of any non-logs file(s).
cli_testing[file_generation][no_file]
Type:
booleanDefault value:
FalseDescription: Activates or disables the generation of any non-logs file(s).
cli_testing[file_generation][hosts]
Type:
booleanDefault value:
TrueDescription: Activates or disables the generation of the hosts file(s).
cli_testing[file_generation][plain]
Type:
booleanDefault value:
TrueDescription: Activates or disables the generation of the RAW file(s). What is meant is a list with only a list of subject (one per line).
cli_testing[file_generation][analytic]
Type:
booleanDefault value:
TrueDescription: Activates or disables the generation of the analytic file(s).
cli_testing[file_generation][unified_results]
Type:
booleanDefault value:
FalseDescription: Activates or disables the generation of the unified results file instead of the splitted one.
cli_testing[file_generation][merge_output_dirs]
Type:
booleanDefault value:
FalseDescription: Activates or disables the merging of the outputs of all inputted files inside a single subdirectory as opposed to the normal behavior.
lookup
Type:
dictDescription: Configures everything related to the lookup tools to use while testing a given subject.
lookup[dns]
Type:
booleanDefault value:
TrueDescription: Activates or disables the usage of the DNS lookup whether possible.
lookup[http_status_code]
Type:
booleanDefault value:
TrueDescription: Activates or disables the usage of the HTTP status code whether possible.
lookup[netinfo]
Type:
booleanDefault value:
TrueDescription: Activates or disables the usage of the network information (or network socket) whether possible.
lookup[special]
Type:
booleanDefault value:
TrueDescription: Activates or disables the usage of our SPECIAL and extra rules whether possible.
lookup[whois]
Type:
booleanDefault value:
TrueDescription: Activates or disables the usage of the WHOIS record (or better said the expiration date in it) whether possible.
lookup[reputation]
Type:
booleanDefault value:
TrueDescription: Activates or disables the usage of the reputation dataset whether possible.
lookup[timeout]
Type:
integerDefault value:
5Description: Sets the default timeout to apply to each lookup utilities everytime it is possible to define a timeout.
lookup[platform]
Type:
booleanDefault value:
TrueDescription: Activates or disables the usage of the Platform API whether possible.
dns
Type:
dictDescription: Configures everything related to the DNS lookup.
dns[server]
Type:
listDefault value:
nullDescription: Sets the DNS server(s) to work with.
Note
When a list is given the following format is expected.
dns_server:
- dns1.example.org
- dns2.example.org
Note
You can specify a port number to use to the DNS server if needed.
As example:
- 127.0.1.53:5353
Warning
Please be careful when you overwrite this option. If one is not correct,
you can almost sure that all results are going to be flagged as
INACTIVE.
dns[protocol]
Type:
stringDefault value:
UDPAvailable values:
UDP,TCP,HTTPS,TLS.Description: Sets the protocol to use for all DNS queries.
dns[follow_server_order]
Type:
booleanDefault value:
TrueDescription: Activates or disables the follow-up of the given order.
dns[trust_server]
Type:
booleanDefault value:
FalseDescription: Activates or disables the trust mode. When the trust mode is active and the first read DNS server gives us a negative response (without any error), we take it as it is.
Otherwise, when the trust mode is disabled, when the first read DNS server gives us a negative response (without any error), we still ask all other DNS servers that were given or found.
dns[delay]
Type:
floatDefault value:
0.0Description: Sets the delay to apply between each DNS query.
proxy
Added in version 4.1.0b12.dev.
Type:
dictDescription: Configures everything related to the proxy settings.
proxy[global]
Added in version 4.1.0b12.dev.
Type:
dictDefault value:
{"http": null, "https": null}
- Description:
The global proxy settings to use when no rules is matching.
The proxy settings matcher works with rules, meaning that it will first try to match any of the given rules against the hostname that is being requested against. If none is matching, it will just take over
proxy[global][http]andproxy[global][https].
Note
Both http and https keys are required. But, if one is missing
and the other one is given, PyFunceble will take over the given one.
If you give the following - as example:
http: http://example.org:8080 https: null
The proxy settings matcher will read/use when no rules is matched:
http: http://example.org:8080 https: http://example.org:8080
proxy[rules]
Added in version 4.1.0b12.dev.
Type:
listDefault value:
[](None)
- Description:
Sets the list of rules the proxy settings matcher has to follow.
The proxy settings matcher will first look at the rules listed under this key.
If you give the following - as example:
proxy: global: http: http://example.de:8080 https: http://example.de:8080 rules: - http: http://example.dev:8080 https: http://example.dev:8080 tld: - com - org - http: socks5h://example.org:8080 https: socks5h://example.org:8080 tld: - onionThe proxy settings matcher will:
use
http://example.dev:8080for anyHTTPandHTTPSrequests to a subject that has thecomandorgTop-Level-Domain (TLD).Example:
example.comandexample.orguse
socks5h://example.org:8080for anyHTTPandHTTPSrequests to a subject that has theonionTop-Level-Domain (TLD).Example:
example.onionWarning
If you explicitly want to use this feature to be able to test
.onionsubjects, be sure to use thesocks5hinstead ofsocks5protocol.
socks5hwill delegate the DNS lookup to the proxy whensocks5will delegate the DNS lookup to the locally defined DNS resolver or server.use
http://example.defor anyHTTPandHTTPSrequests to any other subject that is not matching the previous rules.
user_agent
Type:
dictDescription: Configures the user agent.
user_agent[browser]
Type:
stringDefault value:
chromeAvailable values:
chrome,edge,firefox,ie,opera,safari.Description: Sets the browser to get the get the latest user agent from.
Warning
This option is not taken in consideration if user_agent[custom] is
not set to null.
user_agent[platform]
Type:
stringDefault value:
linuxAvailable values:
linux,macosx,win10Description: Sets the platform to get the get the latest user agent for.
Warning
This option is not taken in consideration if user_agent[custom] is
not set to null.
user_agent[custom]
Type:
stringDefault value:
nullDescription: Sets the user agent to use.
Warning
Setting this index will overwrite the choices made into
user_agent[platform] and user_agent[browser].
http_codes
Type:
dictDescription: Configures everything related to the HTTP status code and the way PyFunceble handles them.
http_codes[self_managed]
Type:
boolDefault value:
FalseDescription: Informs PyFunceble that the status code list should not be managed automatically.
http_codes[list]
Type:
dictDescription: Categorizes the HTTP status codes.
http_codes[list][up]
Type:
listDefault value:
- 100 - 101 - 102 - 200 - 201 - 202 - 203 - 204 - 205 - 206 - 207 - 208 - 226Description: List the HTTP status codes which are considered as
ACTIVE.
http_codes[list][potentially_down]
Type:
listDefault value:
- 400 - 402 - 404 - 409 - 410 - 412 - 414 - 415 - 416 - 451Description: List the HTTP status code which are considered as
INACTIVEorPOTENTIALLY_INACTIVE.
http_codes[list][potentially_up]
Type:
listDefault value:
- 000 - 300 - 301 - 302 - 303 - 304 - 305 - 307 - 308 - 403 - 405 - 406 - 407 - 408 - 411 - 413 - 417 - 418 - 421 - 422 - 423 - 424 - 426 - 428 - 429 - 431 - 500 - 501 - 502 - 503 - 504 - 505 - 506 - 507 - 508 - 510 - 511Description: List the HTTP status code which are considered as
ACTIVEorPOTENTIALLY_ACTIVE.
links
Type:
dictDescription: Sets the list of links which can be used/called by the system when needed.
links[api_date_format]
Type:
stringDefault value:
https://pyfunceble.funilrys.com/api/date-formatDescription: Sets the link to use when we share logs.
links[api_no_referrer]
Type:
stringDefault value:
https://pyfunceble.funilrys.com/api/no-referrerDescription: Sets the link to use when we share logs.
platform
Type:
dictDescription: Configures everything related to the interaction with the Platform API.
platform[push]
Type:
boolDefault value:
FalseDescription: Activates or disables the push of the test datasets to the Platform API.
Warning
This argument is useless if the
PYFUNCEBLE_PLATFORM_APIenvironment variable is not defined.
platform[preferred_status_origin]
Type:
strDefault value:
frequentAvailable values:
frequent,latest,recommendedDescription: Sets the preferred status origin when fetching data from the Platform API.