sravan/system76-edk2
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

.pytool/Plugin: Add CI plugins

Browse Source 
https://bugzilla.tianocore.org/show_bug.cgi?id=2315

Add .pytool directory to the edk2 repository with the
following plugins.  These plugins are in a top level
directory because that can be used with all packages
and platforms.

* CharEncodingCheck
* CompilerPlugin
* DependencyCheck
* DscCompleteCheck
* GuidCheck
* LibraryClassCheck
* SpellCheck

Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
This commit is contained in:
Sean Brogan
2019-10-17 21:40:58 -07:00
committed by Michael D Kinney
parent de4ce46d6e
commit 9da7846c88
22 changed files with 1648 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

127
.pytool/Plugin/SpellCheck/Readme.md Normal file
View File

@@ -0,0 +1,127 @@
# Spell Check Plugin
This CiBuildPlugin scans all the files in a given package and checks for
spelling errors.
This plugin requires NodeJs and cspell. If the plugin doesn't find its required
tools then it will mark the test as skipped.
* NodeJS: https://nodejs.org/en/
* cspell: https://www.npmjs.com/package/cspell
* Src and doc available: https://github.com/streetsidesoftware/cspell
## Configuration
The plugin has a few configuration options to support the UEFI codebase.
``` yaml
"SpellCheck": {
"AuditOnly": False, # If True, log all errors and then mark as skipped
"IgnoreFiles": [], # use gitignore syntax to ignore errors in matching files
"ExtendWords": [], # words to extend to the dictionary for this package
"IgnoreStandardPaths": [], # Standard Plugin defined paths that should be ignore
"AdditionalIncludePaths": [] # Additional paths to spell check (wildcards supported)
}
```
### AuditOnly
Boolean - Default is False.
If True run the test in an Audit only mode which will log all errors but instead
of failing the build it will set the test as skipped. This allows visibility
into the failures without breaking the build.
### IgnoreFiles
This supports .gitignore file and folder matching strings including wildcards
* All files will be parsed regardless but then any spelling errors found within
ignored files will not be reported as an error.
* Errors in ignored files will still be output to the test results as
informational comments.
### ExtendWords
This list allows words to be added to the dictionary for the spell checker when
this package is tested. These follow the rules of the cspell config words field.
### IgnoreStandardPaths
This plugin by default will check the below standard paths. If the package
would like to ignore any of them list that here.
```python
[
# C source
"*.c",
"*.h",
# Assembly files
"*.nasm",
"*.asm",
"*.masm",
"*.s",
# ACPI source language
"*.asl",
# Edk2 build files
"*.dsc", "*.dec", "*.fdf", "*.inf",
# Documentation files
"*.md", "*.txt"
]
```
### AdditionalIncludePaths
If the package would to add additional path patterns to be included in
spellchecking they can be defined here.
## Other configuration
In the cspell.base.json there are numerous other settings configured. There is
no support to override these on a per package basis but future features could
make this available. One interesting configuration option is `minWordLength`.
Currently it is set to _5_ which means all 2,3, and 4 letter words will be
ignored. This helps minimize the number of technical acronyms, register names,
and other UEFI specific values that must be ignored.
## False positives
The cspell dictionary is not perfect and there are cases where technical words
or acronyms are not found in the dictionary. There are three ways to resolve
false positives and the choice for which method should be based on how broadly
the word should be accepted.
### CSpell Base Config file
If the change should apply to all UEFI code and documentation then it should be
added to the base config file `words` section. The base config file is adjacent
to this file and titled `cspell.base.json`. This is a list of accepted words
for all spell checking operations on all packages.
### Package Config
In the package `*.ci.yaml` file there is a `SpellCheck` config section. This
section allows files to be ignored as well as words that should be considered
valid for all files within this package. Add the desired words to the
"ExtendedWords" member.
### In-line File
CSpell supports numerous methods to annotate your files to ignore words,
sections, etc. This can be found in CSpell documentation. Suggestion here is
to use a c-style comment at the top of the file to add words that should be
ignored just for this file. Obviously this has the highest maintenance cost so
it should only be used for file unique words.
``` c
// spell-checker:ignore unenroll, word2, word3
```
or
```ini
# spell-checker:ignore unenroll, word2, word3
```

216
.pytool/Plugin/SpellCheck/SpellCheck.py Normal file
View File

@@ -0,0 +1,216 @@
# @file SpellCheck.py
#
# An edk2-pytool based plugin wrapper for cspell
#
# Copyright (c) Microsoft Corporation.
# SPDX-License-Identifier: BSD-2-Clause-Patent
##
import logging
import json
import yaml
from io import StringIO
import os
from edk2toolext.environment.plugintypes.ci_build_plugin import ICiBuildPlugin
from edk2toollib.utility_functions import RunCmd
from edk2toolext.environment.var_dict import VarDict
from edk2toollib.gitignore_parser import parse_gitignore_lines
from edk2toolext.environment import version_aggregator
class SpellCheck(ICiBuildPlugin):
"""
A CiBuildPlugin that uses the cspell node module to scan the files
from the package being tested for spelling errors. The plugin contains
the base cspell.json file then thru the configuration options other settings
can be changed or extended.
Configuration options:
"SpellCheck": {
"AuditOnly": False, # Don't fail the build if there are errors. Just log them
"IgnoreFiles": [], # use gitignore syntax to ignore errors in matching files
"ExtendWords": [], # words to extend to the dictionary for this package
"IgnoreStandardPaths": [], # Standard Plugin defined paths that should be ignore
"AdditionalIncludePaths": [] # Additional paths to spell check (wildcards supported)
}
"""
#
# A package can remove any of these using IgnoreStandardPaths
#
STANDARD_PLUGIN_DEFINED_PATHS = ["*.c", "*.h",
"*.nasm", "*.asm", "*.masm", "*.s",
"*.asl",
"*.dsc", "*.dec", "*.fdf", "*.inf",
"*.md", "*.txt"
]
def GetTestName(self, packagename: str, environment: VarDict) -> tuple:
""" Provide the testcase name and classname for use in reporting
Args:
packagename: string containing name of package to build
environment: The VarDict for the test to run in
Returns:
a tuple containing the testcase name and the classname
(testcasename, classname)
testclassname: a descriptive string for the testcase can include whitespace
classname: should be patterned <packagename>.<plugin>.<optionally any unique condition>
"""
return ("Spell check files in " + packagename, packagename + ".SpellCheck")
##
# External function of plugin. This function is used to perform the task of the CiBuild Plugin
#
# - package is the edk2 path to package. This means workspace/packagepath relative.
# - edk2path object configured with workspace and packages path
# - PkgConfig Object (dict) for the pkg
# - EnvConfig Object
# - Plugin Manager Instance
# - Plugin Helper Obj Instance
# - Junit Logger
# - output_stream the StringIO output stream from this plugin via logging
def RunBuildPlugin(self, packagename, Edk2pathObj, pkgconfig, environment, PLM, PLMHelper, tc, output_stream=None):
Errors = []
abs_pkg_path = Edk2pathObj.GetAbsolutePathOnThisSytemFromEdk2RelativePath(
packagename)
if abs_pkg_path is None:
tc.SetSkipped()
tc.LogStdError("No package {0}".format(packagename))
return -1
# check for node
return_buffer = StringIO()
ret = RunCmd("node", "--version", outstream=return_buffer)
if (ret != 0):
tc.SetSkipped()
tc.LogStdError("NodeJs not installed. Test can't run")
logging.warning("NodeJs not installed. Test can't run")
return -1
node_version = return_buffer.getvalue().strip() # format vXX.XX.XX
tc.LogStdOut(f"Node version: {node_version}")
version_aggregator.GetVersionAggregator().ReportVersion(
"NodeJs", node_version, version_aggregator.VersionTypes.INFO)
# Check for cspell
return_buffer = StringIO()
ret = RunCmd("cspell", "--version", outstream=return_buffer)
if (ret != 0):
tc.SetSkipped()
tc.LogStdError("cspell not installed. Test can't run")
logging.warning("cspell not installed. Test can't run")
return -1
cspell_version = return_buffer.getvalue().strip() # format XX.XX.XX
tc.LogStdOut(f"CSpell version: {cspell_version}")
version_aggregator.GetVersionAggregator().ReportVersion(
"CSpell", cspell_version, version_aggregator.VersionTypes.INFO)
package_relative_paths_to_spell_check = SpellCheck.STANDARD_PLUGIN_DEFINED_PATHS
#
# Allow the ci.yaml to remove any of the above standard paths
#
if("IgnoreStandardPaths" in pkgconfig):
for a in pkgconfig["IgnoreStandardPaths"]:
if(a in package_relative_paths_to_spell_check):
tc.LogStdOut(
f"ignoring standard path due to ci.yaml ignore: {a}")
package_relative_paths_to_spell_check.remove(a)
else:
tc.LogStdOut(f"Invalid IgnoreStandardPaths value: {a}")
#
# check for any additional include paths defined by package config
#
if("AdditionalIncludePaths" in pkgconfig):
package_relative_paths_to_spell_check.extend(
pkgconfig["AdditionalIncludePaths"])
#
# Make the path string for cspell to check
#
relpath = os.path.relpath(abs_pkg_path)
cpsell_paths = " ".join(
[f"{relpath}/**/{x}" for x in package_relative_paths_to_spell_check])
# Make the config file
config_file_path = os.path.join(
Edk2pathObj.WorkspacePath, "Build", packagename, "cspell_actual_config.json")
mydir = os.path.dirname(os.path.abspath(__file__))
# load as yaml so it can have comments
base = os.path.join(mydir, "cspell.base.yaml")
with open(base, "r") as i:
config = yaml.safe_load(i)
if("ExtendWords" in pkgconfig):
config["words"].extend(pkgconfig["ExtendWords"])
with open(config_file_path, "w") as o:
json.dump(config, o) # output as json so compat with cspell
All_Ignores = []
# Parse the config for other ignores
if "IgnoreFiles" in pkgconfig:
All_Ignores.extend(pkgconfig["IgnoreFiles"])
# spell check all the files
ignore = parse_gitignore_lines(All_Ignores, os.path.join(
abs_pkg_path, "nofile.txt"), abs_pkg_path)
# result is a list of strings like this
# C:\src\sp-edk2\edk2\FmpDevicePkg\FmpDevicePkg.dec:53:9 - Unknown word (Capule)
EasyFix = []
results = self._check_spelling(cpsell_paths, config_file_path)
for r in results:
path, _, word = r.partition(" - Unknown word ")
if len(word) == 0:
# didn't find pattern
continue
pathinfo = path.rsplit(":", 2) # remove the line no info
if(ignore(pathinfo[0])): # check against ignore list
tc.LogStdOut(f"ignoring error due to ci.yaml ignore: {r}")
continue
# real error
EasyFix.append(word.strip().strip("()"))
Errors.append(r)
# Log all errors tc StdError
for l in Errors:
tc.LogStdError(l.strip())
# Helper - Log the syntax needed to add these words to dictionary
if len(EasyFix) > 0:
EasyFix = sorted(set(a.lower() for a in EasyFix))
tc.LogStdOut("\n Easy fix:")
OneString = "If these are not errors add this to your ci.yaml file.\n"
OneString += '"SpellCheck": {\n "ExtendWords": ['
for a in EasyFix:
tc.LogStdOut(f'\n"{a}",')
OneString += f'\n "{a}",'
logging.info(OneString.rstrip(",") + '\n ]\n}')
# add result to test case
overall_status = len(Errors)
if overall_status != 0:
if "AuditOnly" in pkgconfig and pkgconfig["AuditOnly"]:
# set as skipped if AuditOnly
tc.SetSkipped()
return -1
else:
tc.SetFailed("SpellCheck {0} Failed. Errors {1}".format(
packagename, overall_status), "CHECK_FAILED")
else:
tc.SetSuccess()
return overall_status
def _check_spelling(self, abs_file_to_check: str, abs_config_file_to_use: str) -> []:
output = StringIO()
ret = RunCmd(
"cspell", f"--config {abs_config_file_to_use} {abs_file_to_check}", outstream=output)
if ret == 0:
return []
else:
return output.getvalue().strip().splitlines()

11
.pytool/Plugin/SpellCheck/SpellCheck_plug_in.yaml Normal file
View File

@@ -0,0 +1,11 @@
## @file
# CiBuildPlugin used to check spelling
#
# Copyright (c) Microsoft Corporation.
# SPDX-License-Identifier: BSD-2-Clause-Patent
##
{
"scope": "cibuild",
"name": "Spell Check Test",
"module": "SpellCheck"
}

165
.pytool/Plugin/SpellCheck/cspell.base.yaml Normal file
View File

@@ -0,0 +1,165 @@
## @file
# CSpell configuration
#
# Copyright (c) Microsoft Corporation
# SPDX-License-Identifier: BSD-2-Clause-Patent
##
{
"version": "0.1",
"language": "en",
"dictionaries": [
"companies ",
"softwareTerms",
"python",
"cpp"
],
"ignorePaths": [
"*.pdb",
"**/*_extdep/**",
"*.pdf",
"*.exe",
"*.jpg"
],
"minWordLength": 5,
"allowCompoundWords": false,
"ignoreWords": [
"muchange"
],
"words": [
"MTRRs",
"Microarchitecture",
"Goldmont",
"cpuid",
"mwait",
"cstate",
"smram",
"scrtm",
"smbus",
"selftest",
"socket",
"MMRAM",
"qword",
"ENDBR",
"SMBASE",
"FXSAVE",
"FXRSTOR",
"RDRAND",
"IOAPIC",
"ATAPI",
"movsb",
"iretw",
"XENSTORE",
"cdrom",
"oprom",
"oproms",
"varstore",
"EKU",
"ascii",
"nmake",
"NVDIMM",
"nasmb",
"Mtftp",
"Hypercall",
"hypercalls",
"IOMMU",
"QEMU",
"qemus",
"OVMF",
"tiano",
"tianocore",
"edkii",
"coreboot",
"uefipayload",
"bootloader",
"bootloaders",
"mdepkg",
"skuid",
"dxefv",
"toolchain",
"libraryclass",
"preboot",
"pythonpath",
"cygpath",
"nuget",
"basetools",
"prepi",
"OPTEE",
"stringid",
"peims",
"memmap",
"guids",
"uuids",
"smbios",
"certdb",
"certdbv",
"EfiSigList",
"depex",
"IHANDLE",
"Virtio",
"Mbytes",
"Citrix",
"initrd",
"semihost",
"Semihosting",
"Trustzone",
"Fastboot",
"framebuffer",
"genfw",
"TTYTERM",
"miniport",
"LFENCE",
"PCANSI",
"submodule",
"submodules",
"brotli",
"PCCTS",
"softfloat",
"whitepaper",
"ACPICA",
"plugfest",
"bringup",
"formset", #VFR
"ideqvallist",
"numberof",
"oneof",
"endformset",
"endnumeric",
"endoneof",
"disableif",
"guidid",
"classguid",
"efivarstore",
"formsetguid",
"formid",
"suppressif",
"grayoutif",
"ideqval",
"endform",
"endcheckbox",
"questionid",
"questionref",
"enddate",
"endstring",
"guidop",
"endguidop",
"langdef",
"dynamicex",
"tokenspace",
"tokenguid",
"pcd's", #seems like cspell bug
"peim's",
"autogen",
"Disasm",
"Torito",
"SRIOV",
"MRIOV",
"UARTs",
"Consplitter", # common module in UEFI
"FIFOs",
"ACPINVS",
"Endof", # due to of not being uppercase
"bootability",
"Sdhci",
"inmodule",
]
}