Web

Notes on various web-based protocols 

These are my personal notes that I use as a quick help in my work.
You are welcome to read them.

Contents of current page Top-level home page
 
Index  Java Internet Oracle Notes
Linux Basics Web Basics SQL Notes
Informatica Servlets Apache BkpRstore SQL*Plus
Teradata   LDAP Storage PL/SQL
Windows     Tables OEM
UML Lang.   Net8 Portal
SQL Server Python perl Performance OLAP
Vmware Visual Basic PHP/MySQL User Mgmt  
Git        
More technical pages here

Contents

 

Introduction

 

 

ping server.name.com
Can connection be made?
tracert server.name.com
See the route
telnet mail.server.com 25
Test connection
 
 

 

DNS Lookup

nslookup yekesys.com
Gives the cached, non-authoritative answer
nslookup -type=soa yekesys.com
Non-authoritative answer. But take the primary name server, and use in the next step
nslookup -type=soa yekesys.com ns1.ipage.com
This is the authoritative answer. "ns1.ipage.com" is the primary-name-server from the previous step
nslookup -type=MX yekesys.com ns1.ipage.com
Mail server

 

 

 

SMTP

 

 

telnet mail.server.domain 25
--- this opens a telnet session. You may want to log the session

NOOP --> No operation, prevent timeout

HELO local_server.local_domain
   --> or use : EHLO

-- Define the enveloppe with MAIL FROM and RCPT TO
MAIL FROM: my-address@my-server
RCPT TO: dest-address@dest-server NOTIFY=SUCCESS,FAILURE,DELAY

DATA   --> This starts the message (minimum: subject and message)
from: my-address@my-server
to: dest-address@dest-server
subject: test of SMTP
      --> Empty line (CRLF CRLF)
the message goes here
.     --> End with . (CRLF.CRLF)

QUIT

 

Example

telnet mail.server.domain 25
220 mail.server.domain ESMTP Sendmail 1.2.3./4.5.6; Mon, 27 Dec 1999 21:35:02 +0100 (MET)

HELO local_server.local_domain
250 OK

MAIL FROM: my-address@my-server
250 OK - mail from < my-address@my-server >

RCPT TO: dest-address@dest-server NOTIFY=SUCCESS,FAILURE,DELAY
250 dest-address@dest-server... Recipient ok

RCPT TO: dest-address2@dest-server NOTIFY=SUCCESS,FAILURE,DELAY
250 dest-address2@dest-server... Recipient ok

DATA
354 Enter mail, end with "." on a line by itself

from: my-address@my-server
to: dest-address@dest-server
subject: test of SMTP
                  -- empty line
Hi
Is this not wonderful: real bottom-line e-mail?
.                 -- "." on a line by itself
250 VAA20433 Message accepted for delivery

QUIT
221 closing connection

 

Some additional helps

HELP
214-This is Sendmail version 8.9.3
214-Topics:
214- HELO EHLO MAIL RCPT DATA
214- RSET NOOP QUIT HELP VRFY
214- EXPN VERB ETRN DSN
214-For more info use "HELP <TOPIC>".
214-To report bugs in the implementation send email to
214- sendmail-bugs@sendmail.org.
214-For local information send email to Postmaster at your site.
214 End of HELP info

help rcpt
214-RCPT TO: <recipient> [ <parameters> ]
214- Specifies the recipient. Can be used any number of times.
214- Parameters are ESMTP extensions. See "HELP DSN" for details.
214 End of HELP info

help DSN
214-MAIL FROM: <sender> [ RET={ FULL | HDRS} ] [ ENVID=<envid> ]
214-RCPT TO: <recipient> [ NOTIFY={NEVER,SUCCESS,FAILURE,DELAY} ]
214-      [ ORCPT=recipient ]
214- SMTP Delivery Status Notifications.
214-Descriptions:
214- RET    Return either the full message or only headers.
214- ENVID  Sender's "envelope identifier" for tracking.
214- NOTIFY When to send a DSN. Multiple options are OK, comma-
214-        delimited. NEVER must appear by itself.
214- ORCPT Original recipient.
214 End of HELP info

Show name in addition to e-mail address in the FROM or TO fields:
     nice-name <e-mail@address>

 

 

POP3

 

telnet mail.server.domain 110
--- opens a telnet session, you may want to log the session

USER xxxxxx
PASS ******

LIST     --> list all messages
LIST 3   --> just one message
TOP  3 n --> See top n lines
RETR 3   --> retrieve message 3

DELE 3 --> delete message 3

RSET --> undo all deletes

NOOP

QUIT

STAT   --> total messages, with total of bytes
UIDL 3 --> Unique ID for message 3

----------------------- Example -------------

+OK QPOP (version 3.0b18) at rigel starting.
USER the-user-name
+OK Password required for the-user-name.

PASS ***********
+OK the-user-name has 10 messages (4731605 octets).

STAT
+OK 10 4731605

LIST
+OK 9 messages (43080 octets)
1 2545
2 1348
3 24519
4 1041
5 1427
6 1274
7 2476
9 5062
10 3388
.    --> Note that list ends with CRLF.CRLF

TOP 2 1
+OK Message follows
     --- MESSAGE GOES HERE
.    ---------------------> Note that TOP ends with CRLF.CRLF

RETR 2
+OK Message follows
     --- MESSAGE GOES HERE
.    ---------------------> Note that message ends with CRLF.CRLF

NOOP
+OK

DELE 2
+OK Message 2 has been deleted.

QUIT
+OK Pop server at rigel signing off.

 

 

 

File Transfer Protocol (FTP)

 

Common FTP Commands

open machine [port] Open a connection to another machine. User prompt for username/password if -n not set in command line
user username password Define the user and password. Not necessary with auto-login (-n in command line)
close Close connection, equivalent to disconnect
quit Close connection and quit FTP (exit is not a recognized command). Equivalent to bye.
cd Change directory on remote machine
lcd Change directory on local machine
pwd Show path (!pwd for local path)
(m)dir / (m)ls [remote_dir] local_file Type remote directory to local file or show to screen (dash or no local_file). m=multiple.
!dir for local
!pwd    !dir Show local path / local directory contents
ascii / bin Transfer in ascii A (simple text) or binary mode I (e.g. for executables or images)
get / put filename [filename] Get or put one file (eventually change filename)
mget / mput filename* Multiple get or put, using wildcard "*". See "prompt". Remote and local filenames are the same. Interrupt a transfer with Ctrl-c.
prompt Toggle interactive mode between on and off (for mput and mget). See -i command line argument
rename A B Rename on remote host
(m)delete filename Delete on remote machine (m=multiple)
mkdir / rmdir Make a directory on the remote machine / remove directory

 

Other Less-used Commands

append local_file [remote_file] Append local_file to remote_file
bell Beep when command completed
hash Toggle print of one '#' for each data block during transfer
debug Toggle debugging mode (repeat each command preceded with -->). See -d command line argument
glob Toggle metacharacter expansion of local file names. See -g command line argument
literal / quote Send arbitrary string command to remote server. Use with care.
recv / send Receive or send one file (same as get / put)
help / remotehelp Help on local / remote machine
status Show status
trace Toggle packet tracing
type Set transfer type
verbose Toggle verbose mode. See -v command line argument
case Toggle case of filenames. Default:off
cr (UNIX) Toggle strip of CR from CR/LF sequence when receiving records during ASCII type transfers.
size Returns size in bytes
system Shows type of operating system on remote machine

 

Script File

In MS command line: ftp.exe -n -i -s:GET.ftp host > GET.log
In UNIX command line: ftp [-dginptv] [host] > r:\ftp\GET.log

FTP [-v] [-d] [-i] [-n] [-g] [-s:filename] [host]
ftp [-dginptv] [host]

-v Suppresses display of remote server responses.
-n Suppresses auto-login upon initial connection (NT and UNIX).
-i Turns off interactive prompting during multiple file transfers.
-d Enables debugging.
-p Enable pipe as first character (UNIX only)
-t Toggle packet tracing (UNIX)
-g Disables filename globbing (see GLOB command).
-s:filename Specifies a text file containing FTP commands; the commands will automatically run after FTP starts.
host Specifies the host name or IP address of the remote host to connect to.

In the *.ftp file, put:

user username password # yes, the password is visible
cd remote_directory
lcd local_directory
mget / mput / .. # commands
close
quit

 

Look at this when I have a moment:

HOST='the_host'
USER='the_user'
PASSWD='a password'
FILE='the_file_to_send'

ftp -n $HOST << END_SCRIPT
quote USER $USER
quote PASS $PASSWD
cd /usr/d1/target_dir/
lcd /u01/files_to_send/
put $FILE
quit
END_SCRIPT

if [ $? != "0" ]
then
    exit 1
fi

 

More commands

! escape to the shell
bye terminate ftp session and exit
quit terminate ftp session and exit
close terminate ftp session
disconnect terminate ftp session
help print local help information
append append to a file
ascii set ascii transfer type
bell beep when command completed
binary set binary transfer type
cd change remote working directory
delete delete remote file
debug toggle debugging mode
dir list contents of remote directory
get receive file
glob toggle metacharacter expansion of local file names
hash toggle printing `#' for each buffer transferred
lcd change local working directory
cd change remote working directory
delete delete remote file
debug toggle debugging mode
dir list contents of remote directory
get receive file
glob toggle metacharacter expansion of local file names
hash toggle printing `#' for each buffer transferred
lcd change local working directory
cd change remote working directory
literal send arbitrary ftp command
ls nlist contents of remote directory
mdelete delete multiple files
mdir list contents of multiple remote directories
mget get multiple files
mkdir make directory on the remote machine
mls nlist contents of multiple remote directories
mput send multiple files
open connect to remote tftp
prompt force interactive prompting on multiple commands
put send one file
pwd print working directory on remote machine
quote send arbitrary ftp command
recv receive file
remotehelp get help from remote server
rename rename file
rmdir remove directory on the remote machine
send send one file
status * show current status
trace toggle packet tracing
type set file transfer type
user send new user information
verbose toggle verbose mode

 

 

 

 

Secure FTP

 

Passwords are visible when using FTP. They are not visible on the wire with SFTP and SCP. However, they still have to be managed. Public/private key pair authentication removes the need to manage passwords. The next step is VPN tunneling.

 

SFTP = FTP over SSH. Just port 22. Supports SSH keys.
FTPS = FTP over SSL/TSL. Uses several ports in addition to port 21. Supports certificates.

 

SFTP requires knowledge of user name, host name, and (optionally) port. The default port is 22.

cd / lcd Change the directory on the remote server / local computer
chmod, chown Change the permissions / owner on remote server
dir (or ls) / lls List files on remote server / local computer
exit (or quit) Close the connection to the remote computer and exit SFTP
get, put Copy a file
help (or ?) Get help on the use of SFTP commands
mkdir / lmkdir Create a directory on remore server / local computer
ln (or symlink) Create a symbolic link for a file on the remote computer
pwd / lpwd Show the current directory
lumask Change the local umask value
rename Rename a file on the remote host
rm Delete files from the remote computer
rmdir Remove a directory on the remote host (the directory usually has to be empty)
version Display the SFTP version
! In Unix, exit to the shell prompt. Enter exit to return to SFTP. Or follow ! with a command to just execute the command.

 

Set up SSH

Create a public/private key pair on the client
For SSH2: ssh-keygen -b 1024 -t dsa  -f output_keyfile
ssh-keygen -b 1024 -f username_hostname -t dsa
For SSH:  ssh-keygen -b 1024 -t rsa1 -f output_keyfile

When available, prefer SSH2 to SSH.
A password and a file location are needed. The public key is in the file with the extension .pub.

Add the new key to the file of authorized keys:
cat public_key_file.pub >> ~/.ssh/authorized_keys (Some servers only allow SSHv2, meaning that the file should be called authorized_keys2)

Secure the files with chmod go-wrx *.pub

Log in with ssh -i name_of_private_key_file server_name_or_ip

ssh -i .ssh/id_file_name remote_host "ls -l"
scp -i .ssh/id_file_name the_file remote_username@remote_host:the_file
sftp -vb batch_file remote_username@remote_host (-v is for verbose)

The user should own the private key (identity) file and should have exclusive access

With WinSCP / PuTTYgen:

An alternative to SFTP is SCP: scp local_file username@server_name:remote_file

See http://www.linux.com/articles/34958

 

Web Services

Web services can of two styles:

 

Simple Object Access Protocol (SOAP)
Message protocol based on XML; transport via HTTP, FTP, SMTP.
Web Services Description Language (WSDL)
Describes and defines web services using XML. If not present, then the service is not self-describing.
Universal Description, Discovery and Integration (UDDI)
Used to register web services; allows the creation of a web services repository.

SOAP-message:

SOAP Enveloppe

SOAP Header

Header block  

...

Header block  

 

SOAP Body

Body block  

 

Web service techiques:

Web service achitecture

 

XML

 

Xml only has one root element.
Each xml tag must have a closing tag (except for the prolog, which is not technically an xml document).

Prolog:
<?xml version="1.0" encoding="UTF-8"?>
The XML prolog is optional. If it exists, it must be at the beginning of the document.
Put indications for encoding in the prolog.

Two options for empty elements:
<element></element>
<element />
Note that empty elements can still have attributes

It is better to use elements than attributes:

See a structure with freeformatter.com

Namespaces

xmlns="ns..." uses a default namespace ns...
xmlns:x="ns..." x becomes a prefix for the namespace ns....

‘Valid xml’ means that it is validated against an xsd schema.

An xsd definition looks like this: 

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="class">...
</xs:element>
</xs:schema>

This is the default namespace (no prefixes).
Only one default namespace is allowed: any other namespaces have to have a prefix.

<schema xmlns="http://www.w3.org/2001/XMLSchema">
<element name="class">...
</element>
</schema>

Target namespace

<schema targetNamespace="something">
<element name="class">...
</element>
</schema>

Here, the "something" namespace is linked to a file location:

<schema xmlns="something" schemaLocation="something /a/real/file/name something2 /another/FileName2">
<element name="class">...
</element>
</schema>

 

dateTime

DateTime is a primitive data type. Information available at http://www.w3.org/TR/xmlschema-2/#dateTime.
The basic format is: [-]yyyy-mm-ddThh:mm:ss[.ssss][{-|+}hh:mm]. The UTC timezone is either 00:00 or Z.

Some comments:

Examples:
2007-10-08T13:29:00.0000-05:00 (equivalent to 2007-10-08T18:29:00Z)
1962-06-04T13:00:00+01:00 (equivalent to 1962-06-04T12:00:00Z)

DTD

<!ELEMENT x EMPTY> <x/>
<!ELEMENT name (#PCDATA)> <name>a text</name>
<!ELEMENT a_name ANY> <a_name>[any XML data]</a_name>
<!ELEMENT car (gas | diesel)> <car>[gas or diesel but not both]</car>

<!ELEMENT lettre (auteur, destinataire, corps)>
list of elements, in that order
? --> 0 or 1
* --> 0 or many
+ --> 1 or many

<!ELEMENT element (element1, element*)>
--> recursive

 

 

Robots, Crawlers and Other Critters

Un seul fichier "robots.txt" (minuscules) à la racine.

Documentation in wikipedia

User-agent: googlebot
Specifies the agent (robot). Check for requests to robots.txt in the logs.
User-agent: *
All agents (robots)
Disallow: email.htm
Disallow a file
Disallow: /cgi-bin/
Disallow the directory /cgi-bin/
Note that the disallow directive has a wildcard nature, that is "Disallow new" will prevent access to new_file and new_text.
# comment
The "#" must be at the beginning of the line.

Pas de lignes vierges ou blanches car elles sont interpretées comme étant fin de fichier. L'astérisque acceptée uniquement dans le champ User-agent.

 

Examples:

# Allows access to all files and directories (not empty disallow directive)
User-agent: *
Disallow:

 

# Prevents access to all files and directories (note the "/")
User-agent: *
Disallow: /

 

# Prevents access to cgi-bin and images directories:
User-agent: *
Disallow: /cgi-bin/
Disallow: /images/

 

 

 

Lynx

Enter: lynx at command line

H)elp O)ptions P)rint G)o M)ain screen Q)uit /=search [delete]=history lis

g --> enter a URL
up/down keys go from link to link
right arrow to follow link

if link is a file, the d or c options show: d for download, c for cancel
A screen appears with a link to save the file