Difference between pages "Parallelism and Concurrency BoF" and "Tutorials/Tutorial: Packets don't lie: how can you use tcpdump/tshark (wireshark) to prove your point."

From LCA2016 Delegate wiki
(Difference between pages)
Jump to: navigation, search
(Where: add room)
 
(extra)
 
Line 1: Line 1:
We're considering having a Parallelism and Concurrency BoF, for experts and novices alike. It'll most likely be quite informal, with a chance for professionals, enthusiasts and newbies to network and discuss current technologies and application.  
+
===PREREQUISITES===
 +
Please note that this is a tutorial, not a talk.
 +
You should have tcpdump and  wireshark INSTALLED and do some captures BEFORE you come to the tutorial.
 +
To find wireshark packages do something like this:
 +
<pre>
 +
#archlinux
 +
pacman -Ss wireshark tcpdump
 +
#debian
 +
apt-cache search wireshark tcpdump
 +
#fedora
 +
yum search wireshark tcpdump
 +
</pre>
 +
Please install both GUI and CLI packages.
  
== When ==
+
Please add the user you are going to run wireshark/tshark as to the 'wireshark' user group.
 +
<pre>
 +
gpasswd -a james wireshark
 +
</pre>
 +
After this user 'james' will need to log out and log in again!
  
Thursday evening, before the speaker's dinner. Depending on numbers the BoF may retire for a dinner and/or drink at another venue afterwards.
 
  
== Where ==
+
If that still DOESN'T work,  you might want to add a capability. Do it ONLY if you are still unable to do capture.
 +
<pre>
 +
setcap cap_net_raw,cap_net_admin=eip /usr/bin/dumpcap
 +
</pre>
  
D2.193
+
When finished, have a look around at what files came with the package:
 +
<pre>
 +
tcpdump --version
 +
tshark --version
 +
</pre>
 +
List binaries that came with the packge (example is for archlinux, for rmp-like do 'rpm -ql wireshark', for debian 'dpkg -L wireshark')
 +
<pre>
 +
pacman -Ql wireshark-cli|grep bin
 +
wireshark-cli /usr/bin/androiddump
 +
wireshark-cli /usr/bin/capinfos
 +
wireshark-cli /usr/bin/captype        #same as 'capinfos -t'
 +
wireshark-cli /usr/bin/dftest        #display filter byte-code for debugging
 +
wireshark-cli /usr/bin/dumpcap        #can write files
 +
wireshark-cli /usr/bin/editcap        #snaplenth, or split into multiple based on time, number of packets, adjust time
 +
wireshark-cli /usr/bin/idl2wrs
 +
wireshark-cli /usr/bin/mergecap
 +
wireshark-cli /usr/bin/randpkt        #creates a legitimate EthernetII packet with the given Type field set
 +
wireshark-cli /usr/bin/rawshark      #cannot write files, only to standard output
 +
wireshark-cli /usr/bin/reordercap    #part of the functionality of the editcap
 +
wireshark-cli /usr/bin/text2pcap      #hexdump -> pcap
 +
wireshark-cli /usr/bin/tshark
 +
wireshark-cli /usr/include/wireshark/epan/dissectors/packet-ypbind.h
 +
wireshark-cli /usr/share/wireshark/radius/dictionary.bintec
  
== What ==
+
</pre>
  
Parallelism and Concurrency!
+
=Capturing=
 +
====browsing exercise ====
 +
# start capturing
 +
# navigate your browser to linux.conf.au
 +
# navigate your browser to google.com
 +
# navigate your browser to xxxxxxx (your choice)
  
Potential topics are:
+
====have a look at the capture files that you generated====
:Automatic parallelisation / Parallelisation in Declarative Languages
+
<pre>
::[http://www.mercurylang.org/documentation/papers.html#pbone_phd_thesis Paul's Thesis]
+
capinfos -T *.pcap{,ng}
::[http://www.mercurylang.org Mercury Programming Language]
+
</pre>
::[http://www.plasmalang.org Plasma Programming Language] (WIP)
+
==capture interfaces==
:[http://www.intel.com/content/www/us/en/processors/xeon/xeon-phi-detail.html Xeon Phi]
+
<pre>
:OpenMP
+
tcpdump -D
:GPGPU
+
tshark -D
:Software defined radio (SDR)
+
# try with no interface
:Concurrency
+
tshark
:Async IO
+
</pre>
:Map/Reduce
+
:Cloud vs HPC
+
:Async IO
+
:Thread safety, critical sections, mutexes etc.
+
:Python parallelism and concurrency
+
:[http://latproc.github.io/clockwork/# Clockwork] - State based Language with full Concurrency
+
:RCU - What is it?
+
  
Feel free to add more!
+
==capturing on the CLI==
  
== Who is organizing this ==
+
tcpdump can autostop after certain number of packets has been captured. tshark has addinionally duration, filesize and number of files autostop conditions.
 +
What if we want to have a permanent capture running and keep last N days of the logs?
 +
<pre>
 +
tshark -b duration:2 -n -wevery2sec.pcapng & watch -n 1 ls -l
 +
tshark -b duration:2 -n -wevery2sec.pcapng -a files:5  & watch -n1 ls -l
 +
tshark -b duration:3600 -n -weveryHour.pcapng -b files:24
 +
tshark -a filesize:1024 -n -w1MiB.pcapng
 +
</pre>
 +
=expert info=
 +
==GUI==
 +
-r <pcap file>
 +
-J  <jump filter>
 +
          After reading in a capture file using the -r flag, jump to the packet matching the filter (display
 +
          filter syntax).  If no exact match is found the first packet after that is selected.
  
Paul Bone
+
        capinfo
paul at bone dot id dot au
+
        lower bottom corner
  Boney on freenode.org (#linux.conf.au)
+
                expert info
 +
                file name
 +
                packets, etc
 +
        statistics -> protocol hierarchy
 +
        statistics -> HTTP -> packet counter
 +
        statistics -> HTTP -> requests
 +
        extract objects
 +
        follow TCP stream
 +
        coloring rules
 +
        Statistics -> IPv4 Statistics -> Destinations and  Ports
 +
fields
 +
    delta time
 +
   
 +
==CLI==
 +
<pre>
 +
! tshark -q -z ptype,tree
 +
! tshark -q -z io,stat,20,eth -q
 +
! tshark -q -z io,stat,20,http -q
 +
! tshark -q -z io,stat,20,,"BYTES()http" -q
 +
! tshark -q -z http,tree
 +
! tshark -q -z http_req,tree
 +
! tshark -q -z http_srv,tree
 +
create a capture file for icmp
 +
! tshark -q -z icmp,srt
 +
! tshark -q -z io,phs
 +
! tshark -q -z io,stat
 +
! tshark -q -z ip_hosts,tree
 +
! tshark -q -z plen,tree
 +
! tshark -q -z endpoints,eth
 +
! tshark -q -z endpoints,eth,
 +
! tshark -q -z endpoints,ip
 +
! tshark -q -z conv,eth
 +
! tshark -q -z conv,udp
 +
! tshark -q -z conv,tcp
 +
! tshark -q -z conv
 +
! tshark -q -z expert,error -q
 +
! tshark -q -z expert,note -q
 +
</pre>
  
* [[User:Sbragg@deakin.edu.au|Scott Bragg]]
+
=DECRYPTING SSL=
  faulteh_ on freenode.org (#linux.conf.au)
+
When a client (for example, a web browser) makes a connection to a web server requiring SSL/TLS encryption - the encrypted channel is setup using a '''symmetric''' session key. This key is a random string generated by the client and then encrypted and transmitted using the servers '''public''' key, known as the ''Pre-master Secret''. Once shared, the client and server use this shared key to encrypt and decrypt traffic.
  
== Persons Interested in Attending ==
+
MacOS users can do: launchctl setenv SSLKEYLOGFILE /tmp/SSLKEYLOGFILE.txt; open -a Applications/Firefox.app
  
* [[User:Mike.lake@uts.edu.au|Mike Lake]]
+
====exercise on decyphering SSL====
* [https://twitter.com Cera Davies]
+
# <pre>
 +
SSLKEYLOGFILE=/tmp/SSLKEYLOGFILE.txt firefox &  tail -f /tmp/SSLKEYLOGFILE.txt
 +
</pre>
 +
#Navigate to https://google.com
 +
#Do some searches
 +
#Start capture
 +
#Open a new tab and do more searches on google.com
 +
#try https://facebook.com or some other web site.
  
  
{{Template:BoF}}
+
SSLKEYLOGFILE variable  works for firefox, chromium and any program built with NSS library (Network Security Services).
 +
 
 +
==display filter==
 +
<pre>
 +
sack
 +
http
 +
        ip.src==1.1.1.1 &&      tcp.analysis.retransmission or tcp.analysis.fast_retransmission
 +
        http.time >= 0.4
 +
        tcp.analysis.rto >= 0.050
 +
        http.request.uri == "https://www.wireshark.org/"
 +
        http.response.code == 500
 +
        tcp.port in {80 443 8080}
 +
        #the above is same as:
 +
        tcp.port == 80 || tcp.port == 443 || tcp.port == 8080
 +
        _ws.expert.severity >= warn
 +
                0x1      ok
 +
                0x100000 comment
 +
                0x200000 chat
 +
                0x400000 note
 +
                0x600000 warn
 +
                0x800000 error
 +
 
 +
tshark -r /var/tmp/aros.pcapng -e frame.number -e ip.src -e ip.dst -Tfields
 +
tshark -r /srv/http/TCP_SACK.cap  -Y frame.number==29 -V
 +
tshark -r TCP_SACK.cap -Y 'frame.number>=10' -Y 'frame.number<=15'
 +
</pre>
 +
==columns==
 +
<pre>
 +
tshark -r http.pcapng -z follow,tcp,hex,1
 +
tshark -e ip.addr -e tcp.window_size -Tfields
 +
tshark -r http.pcapng -z follow,tcp,hex,127.0.0.1:59544,127.0.0.1:80
 +
tshark -r /srv/http/TCP_SACK.cap  -Tfields -e frame.number -e frame.time_epoch -e ip.src -e ip.dst -e tcp.seq -e tcp.len -e tcp.nxtseq -e tcp.ack  -e tcp.analysis.ack_rtt
 +
tshark -r /srv/http/TCP_SACK.cap  -Tfields -e frame.number -e frame.time_epoch -e ip.src -e ip.dst -e tcp.seq -e tcp.len -e tcp.nxtseq -e tcp.ack  -e  tcp.options.sack_le -e tcp.options.sack_re
 +
</pre>
 +
 
 +
=extra=
 +
<pre>
 +
-d tcp.port==8888,http
 +
</pre>
 +
disable protocols

Latest revision as of 14:34, 3 February 2016

PREREQUISITES

Please note that this is a tutorial, not a talk. You should have tcpdump and wireshark INSTALLED and do some captures BEFORE you come to the tutorial. To find wireshark packages do something like this:

#archlinux
pacman -Ss wireshark tcpdump
#debian
apt-cache search wireshark tcpdump
#fedora
yum search wireshark tcpdump

Please install both GUI and CLI packages.

Please add the user you are going to run wireshark/tshark as to the 'wireshark' user group.

gpasswd -a james wireshark

After this user 'james' will need to log out and log in again!


If that still DOESN'T work, you might want to add a capability. Do it ONLY if you are still unable to do capture.

setcap cap_net_raw,cap_net_admin=eip /usr/bin/dumpcap

When finished, have a look around at what files came with the package:

tcpdump --version
tshark --version

List binaries that came with the packge (example is for archlinux, for rmp-like do 'rpm -ql wireshark', for debian 'dpkg -L wireshark')

pacman -Ql wireshark-cli|grep bin
wireshark-cli /usr/bin/androiddump
wireshark-cli /usr/bin/capinfos
wireshark-cli /usr/bin/captype        #same as 'capinfos -t'
wireshark-cli /usr/bin/dftest         #display filter byte-code for debugging
wireshark-cli /usr/bin/dumpcap        #can write files
wireshark-cli /usr/bin/editcap        #snaplenth, or split into multiple based on time, number of packets, adjust time
wireshark-cli /usr/bin/idl2wrs
wireshark-cli /usr/bin/mergecap
wireshark-cli /usr/bin/randpkt        #creates a legitimate EthernetII packet with the given Type field set
wireshark-cli /usr/bin/rawshark       #cannot write files, only to standard output
wireshark-cli /usr/bin/reordercap     #part of the functionality of the editcap
wireshark-cli /usr/bin/text2pcap      #hexdump -> pcap
wireshark-cli /usr/bin/tshark
wireshark-cli /usr/include/wireshark/epan/dissectors/packet-ypbind.h
wireshark-cli /usr/share/wireshark/radius/dictionary.bintec

Capturing

browsing exercise

  1. start capturing
  2. navigate your browser to linux.conf.au
  3. navigate your browser to google.com
  4. navigate your browser to xxxxxxx (your choice)

have a look at the capture files that you generated

capinfos -T *.pcap{,ng}

capture interfaces

tcpdump -D
tshark -D
# try with no interface
tshark

capturing on the CLI

tcpdump can autostop after certain number of packets has been captured. tshark has addinionally duration, filesize and number of files autostop conditions. What if we want to have a permanent capture running and keep last N days of the logs?

tshark -b duration:2 -n -wevery2sec.pcapng & watch -n 1 ls -l
tshark -b duration:2 -n -wevery2sec.pcapng -a files:5  & watch -n1 ls -l
tshark -b duration:3600 -n -weveryHour.pcapng -b files:24
tshark -a filesize:1024 -n -w1MiB.pcapng

expert info

GUI

-r <pcap file> -J <jump filter>

          After reading in a capture file using the -r flag, jump to the packet matching the filter (display
          filter syntax).  If no exact match is found the first packet after that is selected.
       capinfo
       lower bottom corner
               expert info
               file name
               packets, etc
       statistics -> protocol hierarchy
       statistics -> HTTP -> packet counter
       statistics -> HTTP -> requests
       extract objects
       follow TCP stream
       coloring rules
       Statistics -> IPv4 Statistics ->  Destinations and  Ports

fields

   delta time
   

CLI

! tshark -q -z ptype,tree
! tshark -q -z io,stat,20,eth -q
! tshark -q -z io,stat,20,http -q
! tshark -q -z io,stat,20,,"BYTES()http" -q
! tshark -q -z http,tree
! tshark -q -z http_req,tree
! tshark -q -z http_srv,tree
create a capture file for icmp
! tshark -q -z icmp,srt
! tshark -q -z io,phs
! tshark -q -z io,stat
! tshark -q -z ip_hosts,tree
! tshark -q -z plen,tree
! tshark -q -z endpoints,eth
! tshark -q -z endpoints,eth,
! tshark -q -z endpoints,ip
! tshark -q -z conv,eth
! tshark -q -z conv,udp
! tshark -q -z conv,tcp
! tshark -q -z conv
! tshark -q -z expert,error -q
! tshark -q -z expert,note -q

DECRYPTING SSL

When a client (for example, a web browser) makes a connection to a web server requiring SSL/TLS encryption - the encrypted channel is setup using a symmetric session key. This key is a random string generated by the client and then encrypted and transmitted using the servers public key, known as the Pre-master Secret. Once shared, the client and server use this shared key to encrypt and decrypt traffic.

MacOS users can do: launchctl setenv SSLKEYLOGFILE /tmp/SSLKEYLOGFILE.txt; open -a Applications/Firefox.app

exercise on decyphering SSL

SSLKEYLOGFILE=/tmp/SSLKEYLOGFILE.txt firefox & tail -f /tmp/SSLKEYLOGFILE.txt

  1. Navigate to https://google.com
  2. Do some searches
  3. Start capture
  4. Open a new tab and do more searches on google.com
  5. try https://facebook.com or some other web site.


SSLKEYLOGFILE variable works for firefox, chromium and any program built with NSS library (Network Security Services).

display filter

sack 
http
        ip.src==1.1.1.1 &&      tcp.analysis.retransmission or tcp.analysis.fast_retransmission
        http.time >= 0.4
        tcp.analysis.rto >= 0.050
        http.request.uri == "https://www.wireshark.org/"
        http.response.code == 500
        tcp.port in {80 443 8080}
        #the above is same as:
        tcp.port == 80 || tcp.port == 443 || tcp.port == 8080
        _ws.expert.severity >= warn
                0x1      ok
                0x100000 comment
                0x200000 chat
                0x400000 note
                0x600000 warn
                0x800000 error

tshark -r /var/tmp/aros.pcapng -e frame.number -e ip.src -e ip.dst -Tfields
tshark -r /srv/http/TCP_SACK.cap   -Y frame.number==29 -V
tshark -r TCP_SACK.cap -Y 'frame.number>=10' -Y 'frame.number<=15'

columns

 tshark -r http.pcapng -z follow,tcp,hex,1
 tshark -e ip.addr -e tcp.window_size -Tfields
 tshark -r http.pcapng -z follow,tcp,hex,127.0.0.1:59544,127.0.0.1:80
 tshark -r /srv/http/TCP_SACK.cap   -Tfields -e frame.number -e frame.time_epoch -e ip.src -e ip.dst -e tcp.seq -e tcp.len -e tcp.nxtseq -e tcp.ack  -e tcp.analysis.ack_rtt
 tshark -r /srv/http/TCP_SACK.cap   -Tfields -e frame.number -e frame.time_epoch -e ip.src -e ip.dst -e tcp.seq -e tcp.len -e tcp.nxtseq -e tcp.ack  -e  tcp.options.sack_le -e tcp.options.sack_re

extra

-d tcp.port==8888,http

disable protocols