Help to make iSER working

[Ben RUBSON]

Hello,

I have some difficulties to make iSER (FreeBSD 11 BETA3) working.



### My configuration :

2 servers with Mellanox ConnectX-3 NICs.



### What I did :

I rebuilt kernel adding the following to GENERIC :
options OFED
device mlxen
options COMPAT_LINUXKPI

I built iser :
cd /usr/src/sys/modules/iser/ ; make ; make install ; kldload iser
(I saw later-on that I could have used "device iser" in kernel configuration)

My /etc/ctl.conf :
portal-group pg0 {
discovery-auth-group no-authentication
listen 192.168.2.2
offload iser
}
target iqn.2012-06.com.test:target1 {
auth-group no-authentication
portal-group pg0
alias my-iser-disk
lun 0 {
path /dev/da8
}
}



### Error :

As soon as I do the following :
iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
I get this error :
iscsid[1187]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Invalid argument
iscsid[655]: child process 1187 terminated with exit status 1

"sysctl hw.mlxen1.stat.tx_packets" on initiator shows that no packet is sent at all when trying to connect.

Did I miss something ?

Many thanks !

Ben

[Ben RUBSON]

Hi Max,

Thank you very much for your answer.
Is there any plan to add iSER target to FreeBSD ?

Thank you again,

Ben

Hi Ben,
iSER initiator is supported in FreeBSD-11 but not the iser target.
for the target you should use Linux target such as TGT (user space),
Lio or SCST.
thanks,
Max.

Post by Ben RUBSON
Hello,
I have some difficulties to make iSER (FreeBSD 11 BETA3) working.
2 servers with Mellanox ConnectX-3 NICs.
options OFED
device mlxen
options COMPAT_LINUXKPI
cd /usr/src/sys/modules/iser/ ; make ; make install ; kldload iser
(I saw later-on that I could have used "device iser" in kernel configuration)
portal-group pg0 {
discovery-auth-group no-authentication
listen 192.168.2.2
offload iser
}
target iqn.2012-06.com.test:target1 {
auth-group no-authentication
portal-group pg0
alias my-iser-disk
lun 0 {
path /dev/da8
}
}
iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
iscsid[1187]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Invalid argument
iscsid[655]: child process 1187 terminated with exit status 1
"sysctl hw.mlxen1.stat.tx_packets" on initiator shows that no packet is sent at all when trying to connect.
Did I miss something ?
Many thanks !
Ben
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-scsi

[Max Gurtovoy]

Hi Ben,

Our initial goal was to push the initiator driver to FreeBSD and we
accomplished it.
You are welcome to report your benchmarks running iser and compare to
tcp numbers.
Regarding plans for the target side, it's not on my plate for the near
future (but obviously can be later on :)).
I'm not aware of other contributers plans.

Thanks,
Max.

Post by Ben RUBSON
Hi Max,
Thank you very much for your answer.
Is there any plan to add iSER target to FreeBSD ?
Thank you again,
Ben

Hi Ben,
iSER initiator is supported in FreeBSD-11 but not the iser target.
for the target you should use Linux target such as TGT (user space),
Lio or SCST.
thanks,
Max.

Post by Ben RUBSON
Hello,
I have some difficulties to make iSER (FreeBSD 11 BETA3) working.
2 servers with Mellanox ConnectX-3 NICs.
options OFED
device mlxen
options COMPAT_LINUXKPI
cd /usr/src/sys/modules/iser/ ; make ; make install ; kldload iser
(I saw later-on that I could have used "device iser" in kernel configuration)
portal-group pg0 {
discovery-auth-group no-authentication
listen 192.168.2.2
offload iser
}
target iqn.2012-06.com.test:target1 {
auth-group no-authentication
portal-group pg0
alias my-iser-disk
lun 0 {
path /dev/da8
}
}
iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
iscsid[1187]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Invalid argument
iscsid[655]: child process 1187 terminated with exit status 1
"sysctl hw.mlxen1.stat.tx_packets" on initiator shows that no packet is sent at all when trying to connect.
Did I miss something ?
Many thanks !
Ben
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-scsi

_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-scsi

[Ben RUBSON]

Post by Ben RUBSON
cd /usr/src/sys/modules/iser/ ; make ; make install ; kldload iser
(I saw later-on that I could have used "device iser" in kernel configuration)

I tested the "device iser" in kernel configuration, compilation then fails with an "unknown iser device".
"device iser" can be found in "man iser", but does not seem to be understood by the kernel itself.

Ben

[Ben RUBSON]

Post by Ben RUBSON
iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
iscsid[1187]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Invalid argument

I also get this error (invalid argument) whatever the IP address I give to the command, as if my initiator did not understand the iSER command "ISCSIDCONNECT".
How to be sure everything is correctly compiled with ICL_KERNEL_PROXY enabled ?

Thank you,

Ben

[Ben RUBSON]

I just tested to connect to a Linux iSER enabled target, but I still get this error message at initiator side :
iscsid[1187]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Invalid argument

Post by Ben RUBSON
as if my initiator did not understand the iSER command "ISCSIDCONNECT".
How to be sure everything is correctly compiled with ICL_KERNEL_PROXY enabled ?

Ben

Post by Ben RUBSON
Hi Ben,
iSER initiator is supported in FreeBSD-11 but not the iser target.
for the target you should use Linux target such as TGT (user space),
Lio or SCST.
thanks,
Max.

Post by Ben RUBSON
Hello,
I have some difficulties to make iSER (FreeBSD 11 BETA3) working.
2 servers with Mellanox ConnectX-3 NICs.
options OFED
device mlxen
options COMPAT_LINUXKPI
cd /usr/src/sys/modules/iser/ ; make ; make install ; kldload iser
(I saw later-on that I could have used "device iser" in kernel configuration)
portal-group pg0 {
discovery-auth-group no-authentication
listen 192.168.2.2
offload iser
}
target iqn.2012-06.com.test:target1 {
auth-group no-authentication
portal-group pg0
alias my-iser-disk
lun 0 {
path /dev/da8
}
}
iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
iscsid[1187]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Invalid argument
iscsid[655]: child process 1187 terminated with exit status 1
"sysctl hw.mlxen1.stat.tx_packets" on initiator shows that no packet is sent at all when trying to connect.
Did I miss something ?
Many thanks !
Ben
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-scsi

[Max Gurtovoy]

These are the instructions I get from iscsi maintainer that merged our
code to FreeBSD-11:
1. set WITH_OFED=yes in /etc/make.conf
2. "make buildkernel buildworld installkernel installworld"
3. reboot
4. kldload iser

After that, make sure you configured all the interfaces correctly and
that you have ping between the 2 servers.
later you need to check basic RDMA stuff on your servers (Added
Hans/Oded/Meny to help you if needed).

Only if the above are working you can proceed to check iser.

Max.

Post by Ben RUBSON
iscsid[1187]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Invalid argument

Post by Ben RUBSON
as if my initiator did not understand the iSER command "ISCSIDCONNECT".
How to be sure everything is correctly compiled with ICL_KERNEL_PROXY enabled ?

Ben

Post by Ben RUBSON
Hi Ben,
iSER initiator is supported in FreeBSD-11 but not the iser target.
for the target you should use Linux target such as TGT (user space),
Lio or SCST.
thanks,
Max.

Post by Ben RUBSON
Hello,
I have some difficulties to make iSER (FreeBSD 11 BETA3) working.
2 servers with Mellanox ConnectX-3 NICs.
options OFED
device mlxen
options COMPAT_LINUXKPI
cd /usr/src/sys/modules/iser/ ; make ; make install ; kldload iser
(I saw later-on that I could have used "device iser" in kernel configuration)
portal-group pg0 {
discovery-auth-group no-authentication
listen 192.168.2.2
offload iser
}
target iqn.2012-06.com.test:target1 {
auth-group no-authentication
portal-group pg0
alias my-iser-disk
lun 0 {
path /dev/da8
}
}
iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
iscsid[1187]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Invalid argument
iscsid[655]: child process 1187 terminated with exit status 1
"sysctl hw.mlxen1.stat.tx_packets" on initiator shows that no packet is sent at all when trying to connect.
Did I miss something ?
Many thanks !
Ben
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-scsi

_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-scsi

[Ben RUBSON]

Thank you very much Max for your help.



I managed to compile the kernel with iSER support doing the 3 modifications below.

I added the following to /usr/src/sys/conf/options :
ICL_KERNEL_PROXY opt_global.h

And I added "options ICL_KERNEL_PROXY" to my kernel configuration file.

I then faced this compilation error :
/usr/src/sys/modules/ctl/../../cam/ctl/ctl_frontend_iscsi.c:1358:2: error: implicit declaration of function 'icl_conn_handoff_sock'
icl_conn_handoff_sock(cs->cs_conn, so);
/usr/src/sys/modules/ctl/../../cam/ctl/ctl_frontend_iscsi.c:1358:2: note: did you mean 'icl_soft_handoff_sock'?
/usr/src/sys/dev/iscsi/icl.h:158:7: note: 'icl_soft_handoff_sock' declared here
int icl_soft_handoff_sock(struct icl_conn *ic, struct socket *so);

So I replaced icl_conn_handoff_sock by icl_soft_handoff_sock (sounds like it is the needed correction).

iscsictl -r now does not produce previous "invalid argument" error.

@trasz : perhaps these modifications should be done in FreeBSD 11 source tree ?



Now I'm facing the following issue :

# iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
iscsid[8743]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Input/output error
kernel: ERROR: iser_connect_error: conn 0xfffff8096786f000
iscsid[1078]: child process 8743 terminated with exit status 1

If I remove the "-r" option from the iscsictl command, it correctly connects to the target.

On target side, Lio, targetcli :
/> ls iscsi/
o- iscsi ............................................. [1 Target]
o- iqn.2012-06.com.test:target1 ....................... [1 TPG]
o- tpg1 ........................................... [enabled]
o- acls .......................................... [0 ACLs]
o- luns ........................................... [1 LUN]
| o- lun0 ..................... [iblock/block1 (/dev/sdc7)]
o- portals ..................................... [1 Portal]
o- 192.168.2.2:3260 .................. [OK, iser enabled]

I wanted to make some RDMA connection tests with udaddy for example, but I did not found the FreeBSD version.

Any advice ?

Many thanks !

Ben

Post by Max Gurtovoy
1. set WITH_OFED=yes in /etc/make.conf
2. "make buildkernel buildworld installkernel installworld"
3. reboot
4. kldload iser
After that, make sure you configured all the interfaces correctly and that you have ping between the 2 servers.
later you need to check basic RDMA stuff on your servers (Added Hans/Oded/Meny to help you if needed).
Only if the above are working you can proceed to check iser.
Max.

Post by Ben RUBSON
iscsid[1187]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Invalid argument

Post by Ben RUBSON
as if my initiator did not understand the iSER command "ISCSIDCONNECT".
How to be sure everything is correctly compiled with ICL_KERNEL_PROXY enabled ?

Ben

Post by Ben RUBSON
Hi Ben,
iSER initiator is supported in FreeBSD-11 but not the iser target.
for the target you should use Linux target such as TGT (user space),
Lio or SCST.
thanks,
Max.

Post by Ben RUBSON
Hello,
I have some difficulties to make iSER (FreeBSD 11 BETA3) working.
2 servers with Mellanox ConnectX-3 NICs.
options OFED
device mlxen
options COMPAT_LINUXKPI
cd /usr/src/sys/modules/iser/ ; make ; make install ; kldload iser
(I saw later-on that I could have used "device iser" in kernel configuration)
portal-group pg0 {
discovery-auth-group no-authentication
listen 192.168.2.2
offload iser
}
target iqn.2012-06.com.test:target1 {
auth-group no-authentication
portal-group pg0
alias my-iser-disk
lun 0 {
path /dev/da8
}
}
iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
iscsid[1187]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Invalid argument
iscsid[655]: child process 1187 terminated with exit status 1
"sysctl hw.mlxen1.stat.tx_packets" on initiator shows that no packet is sent at all when trying to connect.
Did I miss something ?
Many thanks !
Ben
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-scsi

_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-scsi

[Max Gurtovoy]

Hi,

Post by Ben RUBSON
Thank you very much Max for your help.
I managed to compile the kernel with iSER support doing the 3 modifications below.
ICL_KERNEL_PROXY opt_global.h
And I added "options ICL_KERNEL_PROXY" to my kernel configuration file.
/usr/src/sys/modules/ctl/../../cam/ctl/ctl_frontend_iscsi.c:1358:2: error: implicit declaration of function 'icl_conn_handoff_sock'
icl_conn_handoff_sock(cs->cs_conn, so);
/usr/src/sys/modules/ctl/../../cam/ctl/ctl_frontend_iscsi.c:1358:2: note: did you mean 'icl_soft_handoff_sock'?
/usr/src/sys/dev/iscsi/icl.h:158:7: note: 'icl_soft_handoff_sock' declared here
int icl_soft_handoff_sock(struct icl_conn *ic, struct socket *so);
So I replaced icl_conn_handoff_sock by icl_soft_handoff_sock (sounds like it is the needed correction).
iscsictl -r now does not produce previous "invalid argument" error.
@trasz : perhaps these modifications should be done in FreeBSD 11 source tree ?
# iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
iscsid[8743]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Input/output error
kernel: ERROR: iser_connect_error: conn 0xfffff8096786f000
iscsid[1078]: child process 8743 terminated with exit status 1

please increase the debug level in both initiator and target and sent it
out.
From first look I think you need to configure the LIO target to work
with demo mode.
Another thing you can try is the user space tgt target (compile with
make ISCSI_RDMA=1) that I work with mostly.

Post by Ben RUBSON
If I remove the "-r" option from the iscsictl command, it correctly connects to the target.
/> ls iscsi/
o- iscsi ............................................. [1 Target]
o- iqn.2012-06.com.test:target1 ....................... [1 TPG]
o- tpg1 ........................................... [enabled]
o- acls .......................................... [0 ACLs]
o- luns ........................................... [1 LUN]
| o- lun0 ..................... [iblock/block1 (/dev/sdc7)]
o- portals ..................................... [1 Portal]
o- 192.168.2.2:3260 .................. [OK, iser enabled]
I wanted to make some RDMA connection tests with udaddy for example, but I did not found the FreeBSD version.
Any advice ?
Many thanks !
Ben

Post by Max Gurtovoy
1. set WITH_OFED=yes in /etc/make.conf
2. "make buildkernel buildworld installkernel installworld"
3. reboot
4. kldload iser
After that, make sure you configured all the interfaces correctly and that you have ping between the 2 servers.
later you need to check basic RDMA stuff on your servers (Added Hans/Oded/Meny to help you if needed).
Only if the above are working you can proceed to check iser.
Max.

Post by Ben RUBSON
iscsid[1187]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Invalid argument

Post by Ben RUBSON
as if my initiator did not understand the iSER command "ISCSIDCONNECT".
How to be sure everything is correctly compiled with ICL_KERNEL_PROXY enabled ?

Ben

Post by Ben RUBSON
Hi Ben,
iSER initiator is supported in FreeBSD-11 but not the iser target.
for the target you should use Linux target such as TGT (user space),
Lio or SCST.
thanks,
Max.

Post by Ben RUBSON
Hello,
I have some difficulties to make iSER (FreeBSD 11 BETA3) working.
2 servers with Mellanox ConnectX-3 NICs.
options OFED
device mlxen
options COMPAT_LINUXKPI
cd /usr/src/sys/modules/iser/ ; make ; make install ; kldload iser
(I saw later-on that I could have used "device iser" in kernel configuration)
portal-group pg0 {
discovery-auth-group no-authentication
listen 192.168.2.2
offload iser
}
target iqn.2012-06.com.test:target1 {
auth-group no-authentication
portal-group pg0
alias my-iser-disk
lun 0 {
path /dev/da8
}
}
iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
iscsid[1187]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Invalid argument
iscsid[655]: child process 1187 terminated with exit status 1
"sysctl hw.mlxen1.stat.tx_packets" on initiator shows that no packet is sent at all when trying to connect.
Did I miss something ?
Many thanks !
Ben
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-scsi

_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-scsi

_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-scsi

[Ben RUBSON]

Post by Ben RUBSON
Thank you very much Max for your help.
I managed to compile the kernel with iSER support doing the 3 modifications below.
ICL_KERNEL_PROXY opt_global.h
And I added "options ICL_KERNEL_PROXY" to my kernel configuration file.
/usr/src/sys/modules/ctl/../../cam/ctl/ctl_frontend_iscsi.c:1358:2: error: implicit declaration of function 'icl_conn_handoff_sock'
icl_conn_handoff_sock(cs->cs_conn, so);
/usr/src/sys/modules/ctl/../../cam/ctl/ctl_frontend_iscsi.c:1358:2: note: did you mean 'icl_soft_handoff_sock'?
/usr/src/sys/dev/iscsi/icl.h:158:7: note: 'icl_soft_handoff_sock' declared here
int icl_soft_handoff_sock(struct icl_conn *ic, struct socket *so);
So I replaced icl_conn_handoff_sock by icl_soft_handoff_sock (sounds like it is the needed correction).
iscsictl -r now does not produce previous "invalid argument" error.
@trasz : perhaps these modifications should be done in FreeBSD 11 source tree ?

I'm also rebuilding world with the following to see if it can help :
# cat /etc/src.conf
WITH_ICL_KERNEL_PROXY="YES"
WITH_OFED="YES"

Post by Ben RUBSON
# iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
iscsid[8743]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Input/output error
kernel: ERROR: iser_connect_error: conn 0xfffff8096786f000
iscsid[1078]: child process 8743 terminated with exit status 1

please increase the debug level in both initiator and target and sent it out.

Logs from the initiator :
iscsid[39668]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Input/output error
kernel: INFO: iser_free_ib_conn_res: freeing conn 0xfffff8048d276000 cma_id 0 qp 0
kernel: DEBUG: iser_conn_connect: before cv_wait: 0xfffff8048d276000
kernel: INFO: iser_cma_handler: event 1 status -19 conn 0xfffff8048d276000 id 0xfffff80880606800
kernel: ERROR: iser_connect_error: conn 0xfffff8048d276000
kernel: DEBUG: iser_conn_connect: after cv_wait: 0xfffff8048d276000
iscsid[1078]: child process 39668 terminated with exit status 1

Logs from the target (here I tried TGT) :
No logs, as if TGT did not receive anything.
If I try to connect without the "-r", here is what I get :
tgtd: login_start(502) Target iqn.2012-06.com.test:target1 is RDMA, but conn cid:0 from iqn.1994-09.org.freebsd:srv1 is TCP
So target is correctly iSER configured.
I tried without iSER on both sides, of course it works.

From first look I think you need to configure the LIO target to work with demo mode.

I double-checked, LIO was in demo mode, I followed this guideline :
https://community.mellanox.com/docs/DOC-1468

One strange thing, on initiator, sounds like no packet goes through the interface when trying to connect :
# sysctl hw.mlxen1.stat.rx_packets hw.mlxen1.stat.tx_packets
hw.mlxen1.stat.rx_packets: 386
hw.mlxen1.stat.tx_packets: 409
# iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
# sysctl hw.mlxen1.stat.rx_packets hw.mlxen1.stat.tx_packets
hw.mlxen1.stat.rx_packets: 386
hw.mlxen1.stat.tx_packets: 409

Another thing you can try is the user space tgt target (compile with make ISCSI_RDMA=1) that I work with mostly.

Post by Ben RUBSON
If I remove the "-r" option from the iscsictl command, it correctly connects to the target.
/> ls iscsi/
o- iscsi ............................................. [1 Target]
o- iqn.2012-06.com.test:target1 ....................... [1 TPG]
o- tpg1 ........................................... [enabled]
o- acls .......................................... [0 ACLs]
o- luns ........................................... [1 LUN]
| o- lun0 ..................... [iblock/block1 (/dev/sdc7)]
o- portals ..................................... [1 Portal]
o- 192.168.2.2:3260 .................. [OK, iser enabled]
I wanted to make some RDMA connection tests with udaddy for example, but I did not found the FreeBSD version.
Any advice ?
Many thanks !
Ben

Post by Max Gurtovoy
1. set WITH_OFED=yes in /etc/make.conf
2. "make buildkernel buildworld installkernel installworld"
3. reboot
4. kldload iser
After that, make sure you configured all the interfaces correctly and that you have ping between the 2 servers.
later you need to check basic RDMA stuff on your servers (Added Hans/Oded/Meny to help you if needed).
Only if the above are working you can proceed to check iser.
Max.

Post by Ben RUBSON
iscsid[1187]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Invalid argument

Post by Ben RUBSON
as if my initiator did not understand the iSER command "ISCSIDCONNECT".
How to be sure everything is correctly compiled with ICL_KERNEL_PROXY enabled ?

Ben

Post by Ben RUBSON
Hi Ben,
iSER initiator is supported in FreeBSD-11 but not the iser target.
for the target you should use Linux target such as TGT (user space),
Lio or SCST.
thanks,
Max.

Post by Ben RUBSON
Hello,
I have some difficulties to make iSER (FreeBSD 11 BETA3) working.
2 servers with Mellanox ConnectX-3 NICs.
options OFED
device mlxen
options COMPAT_LINUXKPI
cd /usr/src/sys/modules/iser/ ; make ; make install ; kldload iser
(I saw later-on that I could have used "device iser" in kernel configuration)
portal-group pg0 {
discovery-auth-group no-authentication
listen 192.168.2.2
offload iser
}
target iqn.2012-06.com.test:target1 {
auth-group no-authentication
portal-group pg0
alias my-iser-disk
lun 0 {
path /dev/da8
}
}
iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
iscsid[1187]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Invalid argument
iscsid[655]: child process 1187 terminated with exit status 1
"sysctl hw.mlxen1.stat.tx_packets" on initiator shows that no packet is sent at all when trying to connect.
Did I miss something ?
Many thanks !
Ben

[Ben RUBSON]

Here are the tests with udaddy from FreeBSD, trying to connect to Linux :

# udaddy -s 192.168.2.2
libibverbs: Warning: couldn't open config directory '/etc/ibverbs/'.
libibverbs: Warning: couldn't open config directory '/etc/ibverbs/'.
udaddy: starting client
udaddy: connecting
libibverbs: Warning: couldn't open config directory '/etc/ibverbs/'.
udaddy: event: RDMA_CM_EVENT_ADDR_ERROR, error: -19
test complete
return status -19

Sounds like the issue is with FreeBSD itself.

Ben

Post by Ben RUBSON

Post by Ben RUBSON
Thank you very much Max for your help.
I managed to compile the kernel with iSER support doing the 3 modifications below.
ICL_KERNEL_PROXY opt_global.h
And I added "options ICL_KERNEL_PROXY" to my kernel configuration file.
/usr/src/sys/modules/ctl/../../cam/ctl/ctl_frontend_iscsi.c:1358:2: error: implicit declaration of function 'icl_conn_handoff_sock'
icl_conn_handoff_sock(cs->cs_conn, so);
/usr/src/sys/modules/ctl/../../cam/ctl/ctl_frontend_iscsi.c:1358:2: note: did you mean 'icl_soft_handoff_sock'?
/usr/src/sys/dev/iscsi/icl.h:158:7: note: 'icl_soft_handoff_sock' declared here
int icl_soft_handoff_sock(struct icl_conn *ic, struct socket *so);
So I replaced icl_conn_handoff_sock by icl_soft_handoff_sock (sounds like it is the needed correction).
iscsictl -r now does not produce previous "invalid argument" error.
@trasz : perhaps these modifications should be done in FreeBSD 11 source tree ?

# cat /etc/src.conf
WITH_ICL_KERNEL_PROXY="YES"
WITH_OFED="YES"

Post by Ben RUBSON
# iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
iscsid[8743]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Input/output error
kernel: ERROR: iser_connect_error: conn 0xfffff8096786f000
iscsid[1078]: child process 8743 terminated with exit status 1

please increase the debug level in both initiator and target and sent it out.

iscsid[39668]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Input/output error
kernel: INFO: iser_free_ib_conn_res: freeing conn 0xfffff8048d276000 cma_id 0 qp 0
kernel: DEBUG: iser_conn_connect: before cv_wait: 0xfffff8048d276000
kernel: INFO: iser_cma_handler: event 1 status -19 conn 0xfffff8048d276000 id 0xfffff80880606800
kernel: ERROR: iser_connect_error: conn 0xfffff8048d276000
kernel: DEBUG: iser_conn_connect: after cv_wait: 0xfffff8048d276000
iscsid[1078]: child process 39668 terminated with exit status 1
No logs, as if TGT did not receive anything.
tgtd: login_start(502) Target iqn.2012-06.com.test:target1 is RDMA, but conn cid:0 from iqn.1994-09.org.freebsd:srv1 is TCP
So target is correctly iSER configured.
I tried without iSER on both sides, of course it works.

From first look I think you need to configure the LIO target to work with demo mode.

https://community.mellanox.com/docs/DOC-1468
# sysctl hw.mlxen1.stat.rx_packets hw.mlxen1.stat.tx_packets
hw.mlxen1.stat.rx_packets: 386
hw.mlxen1.stat.tx_packets: 409
# iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
# sysctl hw.mlxen1.stat.rx_packets hw.mlxen1.stat.tx_packets
hw.mlxen1.stat.rx_packets: 386
hw.mlxen1.stat.tx_packets: 409

Another thing you can try is the user space tgt target (compile with make ISCSI_RDMA=1) that I work with mostly.

Post by Ben RUBSON
If I remove the "-r" option from the iscsictl command, it correctly connects to the target.
/> ls iscsi/
o- iscsi ............................................. [1 Target]
o- iqn.2012-06.com.test:target1 ....................... [1 TPG]
o- tpg1 ........................................... [enabled]
o- acls .......................................... [0 ACLs]
o- luns ........................................... [1 LUN]
| o- lun0 ..................... [iblock/block1 (/dev/sdc7)]
o- portals ..................................... [1 Portal]
o- 192.168.2.2:3260 .................. [OK, iser enabled]
I wanted to make some RDMA connection tests with udaddy for example, but I did not found the FreeBSD version.
Any advice ?
Many thanks !
Ben

Post by Max Gurtovoy
1. set WITH_OFED=yes in /etc/make.conf
2. "make buildkernel buildworld installkernel installworld"
3. reboot
4. kldload iser
After that, make sure you configured all the interfaces correctly and that you have ping between the 2 servers.
later you need to check basic RDMA stuff on your servers (Added Hans/Oded/Meny to help you if needed).
Only if the above are working you can proceed to check iser.
Max.

Post by Ben RUBSON
iscsid[1187]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Invalid argument

Post by Ben RUBSON
as if my initiator did not understand the iSER command "ISCSIDCONNECT".
How to be sure everything is correctly compiled with ICL_KERNEL_PROXY enabled ?

Ben

Post by Ben RUBSON
Hi Ben,
iSER initiator is supported in FreeBSD-11 but not the iser target.
for the target you should use Linux target such as TGT (user space),
Lio or SCST.
thanks,
Max.

Post by Ben RUBSON
Hello,
I have some difficulties to make iSER (FreeBSD 11 BETA3) working.
2 servers with Mellanox ConnectX-3 NICs.
options OFED
device mlxen
options COMPAT_LINUXKPI
cd /usr/src/sys/modules/iser/ ; make ; make install ; kldload iser
(I saw later-on that I could have used "device iser" in kernel configuration)
portal-group pg0 {
discovery-auth-group no-authentication
listen 192.168.2.2
offload iser
}
target iqn.2012-06.com.test:target1 {
auth-group no-authentication
portal-group pg0
alias my-iser-disk
lun 0 {
path /dev/da8
}
}
iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
iscsid[1187]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Invalid argument
iscsid[655]: child process 1187 terminated with exit status 1
"sysctl hw.mlxen1.stat.tx_packets" on initiator shows that no packet is sent at all when trying to connect.
Did I miss something ?
Many thanks !
Ben

[Max Gurtovoy]

Hi,

Post by Ben RUBSON
# udaddy -s 192.168.2.2
libibverbs: Warning: couldn't open config directory '/etc/ibverbs/'.
libibverbs: Warning: couldn't open config directory '/etc/ibverbs/'.
udaddy: starting client
udaddy: connecting
libibverbs: Warning: couldn't open config directory '/etc/ibverbs/'.
udaddy: event: RDMA_CM_EVENT_ADDR_ERROR, error: -19

yes, you got the same event in iser (event 1).
There is a configuration issue with your setup.
Once you overcome this, iSER should work for you.

Meny/Hans,
can you assist here ?

Thanks,
Max.

Post by Ben RUBSON
test complete
return status -19
Sounds like the issue is with FreeBSD itself.
Ben

Post by Ben RUBSON

Post by Ben RUBSON
Thank you very much Max for your help.
I managed to compile the kernel with iSER support doing the 3 modifications below.
ICL_KERNEL_PROXY opt_global.h
And I added "options ICL_KERNEL_PROXY" to my kernel configuration file.
/usr/src/sys/modules/ctl/../../cam/ctl/ctl_frontend_iscsi.c:1358:2: error: implicit declaration of function 'icl_conn_handoff_sock'
icl_conn_handoff_sock(cs->cs_conn, so);
/usr/src/sys/modules/ctl/../../cam/ctl/ctl_frontend_iscsi.c:1358:2: note: did you mean 'icl_soft_handoff_sock'?
/usr/src/sys/dev/iscsi/icl.h:158:7: note: 'icl_soft_handoff_sock' declared here
int icl_soft_handoff_sock(struct icl_conn *ic, struct socket *so);
So I replaced icl_conn_handoff_sock by icl_soft_handoff_sock (sounds like it is the needed correction).
iscsictl -r now does not produce previous "invalid argument" error.
@trasz : perhaps these modifications should be done in FreeBSD 11 source tree ?

# cat /etc/src.conf
WITH_ICL_KERNEL_PROXY="YES"
WITH_OFED="YES"

Post by Ben RUBSON
# iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
iscsid[8743]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Input/output error
kernel: ERROR: iser_connect_error: conn 0xfffff8096786f000
iscsid[1078]: child process 8743 terminated with exit status 1

please increase the debug level in both initiator and target and sent it out.

iscsid[39668]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Input/output error
kernel: INFO: iser_free_ib_conn_res: freeing conn 0xfffff8048d276000 cma_id 0 qp 0
kernel: DEBUG: iser_conn_connect: before cv_wait: 0xfffff8048d276000
kernel: INFO: iser_cma_handler: event 1 status -19 conn 0xfffff8048d276000 id 0xfffff80880606800
kernel: ERROR: iser_connect_error: conn 0xfffff8048d276000
kernel: DEBUG: iser_conn_connect: after cv_wait: 0xfffff8048d276000
iscsid[1078]: child process 39668 terminated with exit status 1
No logs, as if TGT did not receive anything.
tgtd: login_start(502) Target iqn.2012-06.com.test:target1 is RDMA, but conn cid:0 from iqn.1994-09.org.freebsd:srv1 is TCP
So target is correctly iSER configured.
I tried without iSER on both sides, of course it works.

From first look I think you need to configure the LIO target to work with demo mode.

https://community.mellanox.com/docs/DOC-1468
# sysctl hw.mlxen1.stat.rx_packets hw.mlxen1.stat.tx_packets
hw.mlxen1.stat.rx_packets: 386
hw.mlxen1.stat.tx_packets: 409
# iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
# sysctl hw.mlxen1.stat.rx_packets hw.mlxen1.stat.tx_packets
hw.mlxen1.stat.rx_packets: 386
hw.mlxen1.stat.tx_packets: 409

Another thing you can try is the user space tgt target (compile with make ISCSI_RDMA=1) that I work with mostly.

Post by Ben RUBSON
If I remove the "-r" option from the iscsictl command, it correctly connects to the target.
/> ls iscsi/
o- iscsi ............................................. [1 Target]
o- iqn.2012-06.com.test:target1 ....................... [1 TPG]
o- tpg1 ........................................... [enabled]
o- acls .......................................... [0 ACLs]
o- luns ........................................... [1 LUN]
| o- lun0 ..................... [iblock/block1 (/dev/sdc7)]
o- portals ..................................... [1 Portal]
o- 192.168.2.2:3260 .................. [OK, iser enabled]
I wanted to make some RDMA connection tests with udaddy for example, but I did not found the FreeBSD version.
Any advice ?
Many thanks !
Ben

Post by Max Gurtovoy
1. set WITH_OFED=yes in /etc/make.conf
2. "make buildkernel buildworld installkernel installworld"
3. reboot
4. kldload iser
After that, make sure you configured all the interfaces correctly and that you have ping between the 2 servers.
later you need to check basic RDMA stuff on your servers (Added Hans/Oded/Meny to help you if needed).
Only if the above are working you can proceed to check iser.
Max.

Post by Ben RUBSON
iscsid[1187]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Invalid argument

Post by Ben RUBSON
as if my initiator did not understand the iSER command "ISCSIDCONNECT".
How to be sure everything is correctly compiled with ICL_KERNEL_PROXY enabled ?

Ben

Post by Ben RUBSON
Hi Ben,
iSER initiator is supported in FreeBSD-11 but not the iser target.
for the target you should use Linux target such as TGT (user space),
Lio or SCST.
thanks,
Max.

Post by Ben RUBSON
Hello,
I have some difficulties to make iSER (FreeBSD 11 BETA3) working.
2 servers with Mellanox ConnectX-3 NICs.
options OFED
device mlxen
options COMPAT_LINUXKPI
cd /usr/src/sys/modules/iser/ ; make ; make install ; kldload iser
(I saw later-on that I could have used "device iser" in kernel configuration)
portal-group pg0 {
discovery-auth-group no-authentication
listen 192.168.2.2
offload iser
}
target iqn.2012-06.com.test:target1 {
auth-group no-authentication
portal-group pg0
alias my-iser-disk
lun 0 {
path /dev/da8
}
}
iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
iscsid[1187]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Invalid argument
iscsid[655]: child process 1187 terminated with exit status 1
"sysctl hw.mlxen1.stat.tx_packets" on initiator shows that no packet is sent at all when trying to connect.
Did I miss something ?
Many thanks !
Ben

_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-scsi

[Hans Petter Selasky]

Hi,

Error 19 means ENODEV. Possibly something is not supported.

Meny: I wonder if it is better if Ben install our infiniband patches to bring the IB code up to v3.2?

--HPS

-----Original Message-----
From: Max Gurtovoy
Sent: Friday, August 12, 2016 1:51 AM
To: Ben RUBSON <***@gmail.com>; freebsd-***@freebsd.org
Cc: Oren Duer <***@mellanox.com>; Oded Shanoon <***@mellanox.com>; Meny Yossefi <***@mellanox.com>; Hans Petter Selasky <***@mellanox.com>; Roy Shterman <***@mellanox.com>
Subject: Re: Help to make iSER working

Hi,

Post by Ben RUBSON
# udaddy -s 192.168.2.2
libibverbs: Warning: couldn't open config directory '/etc/ibverbs/'.
libibverbs: Warning: couldn't open config directory '/etc/ibverbs/'.
udaddy: starting client
udaddy: connecting
libibverbs: Warning: couldn't open config directory '/etc/ibverbs/'.
udaddy: event: RDMA_CM_EVENT_ADDR_ERROR, error: -19

yes, you got the same event in iser (event 1).
There is a configuration issue with your setup.
Once you overcome this, iSER should work for you.

Meny/Hans,
can you assist here ?

Thanks,
Max.

Post by Ben RUBSON
test complete
return status -19
Sounds like the issue is with FreeBSD itself.
Ben

Post by Ben RUBSON

Post by Ben RUBSON
Thank you very much Max for your help.
I managed to compile the kernel with iSER support doing the 3 modifications below.
ICL_KERNEL_PROXY opt_global.h
And I added "options ICL_KERNEL_PROXY" to my kernel configuration file.
/usr/src/sys/modules/ctl/../../cam/ctl/ctl_frontend_iscsi.c:1358:2: error: implicit declaration of function 'icl_conn_handoff_sock'
icl_conn_handoff_sock(cs->cs_conn, so);
/usr/src/sys/modules/ctl/../../cam/ctl/ctl_frontend_iscsi.c:1358:2: note: did you mean 'icl_soft_handoff_sock'?
/usr/src/sys/dev/iscsi/icl.h:158:7: note: 'icl_soft_handoff_sock' declared here
int icl_soft_handoff_sock(struct icl_conn *ic, struct socket *so);
So I replaced icl_conn_handoff_sock by icl_soft_handoff_sock (sounds like it is the needed correction).
iscsictl -r now does not produce previous "invalid argument" error.
@trasz : perhaps these modifications should be done in FreeBSD 11 source tree ?

# cat /etc/src.conf
WITH_ICL_KERNEL_PROXY="YES"
WITH_OFED="YES"

Post by Ben RUBSON
# iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
iscsid[8743]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to
Input/output error
kernel: ERROR: iser_connect_error: conn 0xfffff8096786f000
iscsid[1078]: child process 8743 terminated with exit status 1

please increase the debug level in both initiator and target and sent it out.

iscsid[39668]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to
Input/output error
kernel: INFO: iser_free_ib_conn_res: freeing conn 0xfffff8048d276000 cma_id 0 qp 0
kernel: DEBUG: iser_conn_connect: before cv_wait: 0xfffff8048d276000
kernel: INFO: iser_cma_handler: event 1 status -19 conn
0xfffff8048d276000 id 0xfffff80880606800
kernel: ERROR: iser_connect_error: conn 0xfffff8048d276000
kernel: DEBUG: iser_conn_connect: after cv_wait: 0xfffff8048d276000
iscsid[1078]: child process 39668 terminated with exit status 1
No logs, as if TGT did not receive anything.
tgtd: login_start(502) Target iqn.2012-06.com.test:target1 is RDMA,
but conn cid:0 from iqn.1994-09.org.freebsd:srv1 is TCP So target is correctly iSER configured.
I tried without iSER on both sides, of course it works.

From first look I think you need to configure the LIO target to work with demo mode.

https://community.mellanox.com/docs/DOC-1468
# sysctl hw.mlxen1.stat.rx_packets hw.mlxen1.stat.tx_packets
hw.mlxen1.stat.rx_packets: 386
hw.mlxen1.stat.tx_packets: 409
# iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2 #
sysctl hw.mlxen1.stat.rx_packets hw.mlxen1.stat.tx_packets
hw.mlxen1.stat.rx_packets: 386
hw.mlxen1.stat.tx_packets: 409

Another thing you can try is the user space tgt target (compile with make ISCSI_RDMA=1) that I work with mostly.

Post by Ben RUBSON
If I remove the "-r" option from the iscsictl command, it correctly connects to the target.
/> ls iscsi/
o- iscsi ............................................. [1 Target]
o- iqn.2012-06.com.test:target1 ....................... [1 TPG]
o- tpg1 ........................................... [enabled]
o- acls .......................................... [0 ACLs]
o- luns ........................................... [1 LUN]
| o- lun0 ..................... [iblock/block1 (/dev/sdc7)]
o- portals ..................................... [1 Portal]
o- 192.168.2.2:3260 .................. [OK, iser enabled]
I wanted to make some RDMA connection tests with udaddy for example, but I did not found the FreeBSD version.
Any advice ?
Many thanks !
Ben

1. set WITH_OFED=yes in /etc/make.conf 2. "make buildkernel
buildworld installkernel installworld"
3. reboot
4. kldload iser
After that, make sure you configured all the interfaces correctly and that you have ping between the 2 servers.
later you need to check basic RDMA stuff on your servers (Added Hans/Oded/Meny to help you if needed).
Only if the above are working you can proceed to check iser.
Max.

Post by Ben RUBSON
iscsid[1187]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed
Invalid argument

Post by Ben RUBSON
as if my initiator did not understand the iSER command "ISCSIDCONNECT".
How to be sure everything is correctly compiled with ICL_KERNEL_PROXY enabled ?

Ben

Post by Ben RUBSON
Hi Ben,
iSER initiator is supported in FreeBSD-11 but not the iser target.
for the target you should use Linux target such as TGT (user
space), Lio or SCST.
thanks,
Max.

Post by Ben RUBSON
Hello,
I have some difficulties to make iSER (FreeBSD 11 BETA3) working.
2 servers with Mellanox ConnectX-3 NICs.
options OFED
device mlxen
options COMPAT_LINUXKPI
cd /usr/src/sys/modules/iser/ ; make ; make install ; kldload
iser (I saw later-on that I could have used "device iser" in
kernel configuration)
portal-group pg0 {
discovery-auth-group no-authentication listen 192.168.2.2
offload iser } target iqn.2012-06.com.test:target1 { auth-group
no-authentication portal-group pg0 alias my-iser-disk lun 0 {
path /dev/da8 } }
iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2 I
ISCSIDCONNECT: Invalid argument
iscsid[655]: child process 1187 terminated with exit status 1
"sysctl hw.mlxen1.stat.tx_packets" on initiator shows that no packet is sent at all when trying to connect.
Did I miss something ?
Many thanks !
Ben

_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-scsi

[Ben RUBSON]

Hi,

I went with a fresh new install to have a clean status.

What I did :

// FreeBSD 11-RC1 installation
# echo "WITH_OFED=YES" > /etc/make.conf
# cd /usr/src/ ; make buildkernel KERNCONF=GENERIC ; make installkernel KERNCONF=GENERIC
# cd /usr/src/sys/modules/mlx4 ; make ; make install
# cd /usr/src/sys/modules/mlxen ; make ; make install
# cd /usr/src/sys/modules/iser ; make ; make install
# echo "mlx4_load=YES" >> /boot/loader.conf
# echo "mlxen_load=YES" >> /boot/loader.conf
# echo "iser_load=YES" >> /boot/loader.conf
# reboot

Everything went fine, no code modition needed or whatever, perfect.

# iscsictl -A -t iqn.2012-06.com.test:target1 -p 192.168.2.2
On the target, I get :
tgtd[1912]: tgtd: login_start(502) Target iqn.2012-06.com.test:target1 is RDMA, but conn cid:0 from iqn.1994-09.org.freebsd:srv1 is TCP
Perfect, the target is correctly iSER enabled.

# iscsictl -A -t iqn.2012-06.com.test:target1 -p 192.168.2.2 -r
I still get the following error :
iscsid[1062]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Input/output error
kernel: INFO: iser_free_ib_conn_res: freeing conn 0xfffff80147771000 cma_id 0 qp 0
kernel: DEBUG: iser_conn_connect: before cv_wait: 0xfffff80147771000
kernel: INFO: iser_cma_handler: event 1 status -19 conn 0xfffff80147771000 id 0xfffff8005f0e5400
kernel: ERROR: iser_connect_error: conn 0xfffff80147771000
kernel: DEBUG: iser_conn_connect: after cv_wait: 0xfffff80147771000
iscsid[853]: child process 1062 terminated with exit status 1

Waiting for your instructions, would really be nice to have it working for 11 release.

Many thanks !

Best regards,

Ben

Post by Hans Petter Selasky
Hi,
Error 19 means ENODEV. Possibly something is not supported.
Meny: I wonder if it is better if Ben install our infiniband patches to bring the IB code up to v3.2?
--HPS
-----Original Message-----
From: Max Gurtovoy
Sent: Friday, August 12, 2016 1:51 AM
Subject: Re: Help to make iSER working
Hi,

Post by Ben RUBSON
# udaddy -s 192.168.2.2
libibverbs: Warning: couldn't open config directory '/etc/ibverbs/'.
libibverbs: Warning: couldn't open config directory '/etc/ibverbs/'.
udaddy: starting client
udaddy: connecting
libibverbs: Warning: couldn't open config directory '/etc/ibverbs/'.
udaddy: event: RDMA_CM_EVENT_ADDR_ERROR, error: -19

yes, you got the same event in iser (event 1).
There is a configuration issue with your setup.
Once you overcome this, iSER should work for you.
Meny/Hans,
can you assist here ?
Thanks,
Max.

[Ben RUBSON]

Post by Ben RUBSON
Hi,
I went with a fresh new install to have a clean status.
// FreeBSD 11-RC1 installation
# echo "WITH_OFED=YES" > /etc/make.conf
# cd /usr/src/ ; make buildkernel KERNCONF=GENERIC ; make installkernel KERNCONF=GENERIC
# cd /usr/src/sys/modules/mlx4 ; make ; make install
# cd /usr/src/sys/modules/mlxen ; make ; make install
# cd /usr/src/sys/modules/iser ; make ; make install
# echo "mlx4_load=YES" >> /boot/loader.conf
# echo "mlxen_load=YES" >> /boot/loader.conf
# echo "iser_load=YES" >> /boot/loader.conf
# reboot

This was missing :
cd /usr/src/sys/modules/mlx4ib/ ; make ; make install ; kldload mlx4ib.ko

Now it works perfectly...
I'm really sorry for the useless noise on this list...

I hope this above howto will help others.

Thank you again for your support, and sorry again :-/

Benchmarks will of course follow.

Best regards,

Ben

[Ben RUBSON]

Post by Ben RUBSON
Benchmarks will of course follow.

(also posted @freebsd-fs)

I then used Linux TGT as a test target.

### Target : local disk throughput, one thread :
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes (11 GB) copied, 21.3898 s, 502 MB/s

### Initiator : iscsi disk throughput, one thread :
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes transferred in 34.938676 secs (307321843 bytes/sec) - 293 MB/s

### Initiator : iSER disk throughput, one thread :
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes transferred in 20.371947 secs (527068838 bytes/sec) - 502 MB/s

In my example, iSER allows to reach the raw disk throughput (on one thread), whereas iSCSI/TCP is a bottleneck.

I also tested with a target RAM device :

### Initiator : iscsi RAM throughput, one thread :
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes transferred in 24.194665 secs (443792803 bytes/sec) - 423 MB/s

### Initiator : iSER RAM throughput, one thread :
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes transferred in 10.991094 secs (976919873 bytes/sec) - 932 MB/s
(just don't know what is the bottleneck here, of course not the RAM, and network on one thread is around 30GB/s)

Post by Ben RUBSON
Regarding plans for the target side, it's not on my plate for the near future (but obviously can be later on :)).
I'm not aware of other contributers plans.

Can't wait to see this then :)

Many thanks again !

Best regards,

Ben

[Max Gurtovoy]

Good to see that you got a nice progress there.
You can try the fio tool and see even better performance.
You can also use Mellanox community site to see how to reach wire spead
and leaverage your numbers. The iser posts are mostly for linux users
but you can find it helpfull for your needs to.

Thanks,
Max.

Post by Ben RUBSON

Post by Ben RUBSON
Benchmarks will of course follow.

I then used Linux TGT as a test target.
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes (11 GB) copied, 21.3898 s, 502 MB/s
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes transferred in 34.938676 secs (307321843 bytes/sec) - 293 MB/s
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes transferred in 20.371947 secs (527068838 bytes/sec) - 502 MB/s
In my example, iSER allows to reach the raw disk throughput (on one thread), whereas iSCSI/TCP is a bottleneck.
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes transferred in 24.194665 secs (443792803 bytes/sec) - 423 MB/s
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes transferred in 10.991094 secs (976919873 bytes/sec) - 932 MB/s
(just don't know what is the bottleneck here, of course not the RAM, and network on one thread is around 30GB/s)

Post by Ben RUBSON
Regarding plans for the target side, it's not on my plate for the near future (but obviously can be later on :)).
I'm not aware of other contributers plans.

Can't wait to see this then :)
Many thanks again !
Best regards,
Ben

[Ben RUBSON]

Thank you very much Max for all your advices.

These tests were really interesting as they demonstrated that iSER can easily improve my iSCSI storage infrastructure throughput.

Unfortunately, for production purpose, I need FreeBSD as the target system.
I will then keep an eye on future FreeBSD releases related to iSER.

Thank you again !

Ben

Post by Max Gurtovoy
Good to see that you got a nice progress there.
You can try the fio tool and see even better performance.
You can also use Mellanox community site to see how to reach wire spead and leaverage your numbers. The iser posts are mostly for linux users but you can find it helpfull for your needs to.
Thanks,
Max.

Post by Ben RUBSON

Post by Ben RUBSON
Benchmarks will of course follow.

I then used Linux TGT as a test target.
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes (11 GB) copied, 21.3898 s, 502 MB/s
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes transferred in 34.938676 secs (307321843 bytes/sec) - 293 MB/s
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes transferred in 20.371947 secs (527068838 bytes/sec) - 502 MB/s
In my example, iSER allows to reach the raw disk throughput (on one thread), whereas iSCSI/TCP is a bottleneck.
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes transferred in 24.194665 secs (443792803 bytes/sec) - 423 MB/s
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes transferred in 10.991094 secs (976919873 bytes/sec) - 932 MB/s
(just don't know what is the bottleneck here, of course not the RAM, and network on one thread is around 30GB/s)

Post by Ben RUBSON
Regarding plans for the target side, it's not on my plate for the near future (but obviously can be later on :)).
I'm not aware of other contributers plans.

Can't wait to see this then :)
Many thanks again !
Best regards,
Ben

[Max Gurtovoy]

NP.

I never tried it on FreeBSD but you can try to build the user space
target TGT on it.
Maybe you'll need some adaptations but should be easier that adding a
new iser target driver from scratch.

Post by Ben RUBSON
Thank you very much Max for all your advices.
These tests were really interesting as they demonstrated that iSER can easily improve my iSCSI storage infrastructure throughput.
Unfortunately, for production purpose, I need FreeBSD as the target system.
I will then keep an eye on future FreeBSD releases related to iSER.
Thank you again !
Ben

Post by Max Gurtovoy
Good to see that you got a nice progress there.
You can try the fio tool and see even better performance.
You can also use Mellanox community site to see how to reach wire spead and leaverage your numbers. The iser posts are mostly for linux users but you can find it helpfull for your needs to.
Thanks,
Max.

Post by Ben RUBSON

Post by Ben RUBSON
Benchmarks will of course follow.

I then used Linux TGT as a test target.
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes (11 GB) copied, 21.3898 s, 502 MB/s
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes transferred in 34.938676 secs (307321843 bytes/sec) - 293 MB/s
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes transferred in 20.371947 secs (527068838 bytes/sec) - 502 MB/s
In my example, iSER allows to reach the raw disk throughput (on one thread), whereas iSCSI/TCP is a bottleneck.
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes transferred in 24.194665 secs (443792803 bytes/sec) - 423 MB/s
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes transferred in 10.991094 secs (976919873 bytes/sec) - 932 MB/s
(just don't know what is the bottleneck here, of course not the RAM, and network on one thread is around 30GB/s)

Post by Ben RUBSON
Regarding plans for the target side, it's not on my plate for the near future (but obviously can be later on :)).
I'm not aware of other contributers plans.

Can't wait to see this then :)
Many thanks again !
Best regards,
Ben