[Hinty] Add type hints to Scapy #2158

gpotter2 · 2019-07-21T07:31:32Z

On hold

please wait before starting anything. There will be a major update to Hinty to tackle type hinting of the core (at least fields & packet). This will allow contributors to tackle smaller parts (the layers). In the meantime, have a look at the other contributions wanted page: #399 - thanks

Project "Hinty" aims at adding Type hints to Scapy. It will help discover bugs, improve the API, and make Scapy up-to-date with the high standards of Python libraries.

Implementation

We use mypy to ensure automatic testing of the work that has already been completed. PRs that fall under project Hinty will process one (or a few) files and register them into the checks. The file .config/mypy/mypy_enabled.txt lists the files in which mypy checks are enabled. This process has been added by #2162

Because we support Python 2.7, the format of the Type hints should be the following:
https://mypy.readthedocs.io/en/latest/cheat_sheet.html

Guide on how to contribute

Pick a file you want to work on.
Add that file to .config/mypy/mypy_enabled.txt
pip install pyannotate cryptography tox
Run the tests for your file: ./test/run_tests -x -c test/configs/[X].utsc -K tcpdump -K manufdb -K wireshark -K tshark. Where [X] matches your OS: linux, windows or osx
This will create test/pyannotate_results
pyannotate --type-info test/pyannotate_results -w [the file.py you are working on]
The file has been automatically processed. Now, edit it to fix the mistakes, and check your work with tox -e mypy.
- Remember that you can use cast(type, obj) from typing to help mypy without affecting the code.
- Import any types you might need from scapy.compat instead of typing. It provides a fallback if typing isn't installed (python < 3.5)
If you get error: No library stub file for module ..., add an exception for it in .config/mypy/mypy.ini (follow the existing format)
Check that the files still pass PEP8: tox -e flake8
When everything passes, submit a PR. 😄

Status

The numbers correspond to the ammount of files processed
MyPy Support: 1.51%

[main]: 9.09%
asn1: 0.00%
layers: 0.00%
libs: 0.00%
contrib: 0.87%
arch: 0.00%
tools: 0.00%
modules: 0.00%


        secdev#2158 - Add Hinty for plist.py


        #2158 - Add Hinty for plist.py - Adds mypy from Issue #2158 (#2358)

* #2158 - Add Hinty for plist.py * Import MATPLOTLIB from extlib.py * Add Line2D and update for review comments * Add regression test changes for new matplotlib.lines import * Remove unnecessary space * Add new line to restart appveyer build

fennellkyle · 2020-05-06T08:51:52Z

I'd like to help but am running into a couple issues.
UTscapy is in scapy/scapy/tools and from there it cannot import scapy.

Traceback (most recent call last):
  File "UTscapy.py", line 27, in <module>
    from scapy.consts import WINDOWS
ModuleNotFoundError: No module named 'scapy'

If I move it to the project root it runs fine now, but pyannotate_results will be in the project root and not in test/ as the 4th step suggests: pyannotate --type-info test/pyannotate_results -w [the file.py you are working on].
Not a show stopper, I just change the path. But then pyannotate errors out:
pyannotate_tools.annotations.parse.ParseError: Invalid type comment: (type, *type) -> scapy.modules.six:with_metaclass.<locals>.metaclass

Can you offer guidance?
Would love to contribute as much as I can.

gpotter2 · 2020-05-26T14:24:16Z

Thanks for the note. I've updated the guide.

There is a bug with pyannotate: you have to manually search + replace all <locals> in the results file with (anything really, blank or whatever. It can't parse the < and this won't work anyways since it's a local).

fennellkyle · 2020-05-28T14:37:56Z

Thanks. There is no test/configs/osx.utsc 😞 as noted. I tried using linux.utsc instead on my Mac but it hangs shortly after ### Loading: test/tftp.uts and doesn't complete...

gpotter2 · 2020-05-28T14:38:48Z

OSX is based on BSD 👍 there is a bsd.uts

gpotter2 added cleanup contributions wanted discussion labels Jul 21, 2019

gpotter2 changed the title ~~Sanitary cleanup~~ Docstrings sanitary cleanup Jul 21, 2019