Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

Commit 70c96ca

Browse files
syurkevi9prady9
andauthored
change setup.py to optionally build binaries in source distribution (#245)
* adds initial skbuild configuration load from local lib directory loads local binaries if they are available * remove git from toml dependencies * additional cmake args in setup.py * add description format for pypi * directory corrections for skbuild * load local nvrtc-builtins when possible * make mkl libs static for distribution * Update README to include information about wheels * remove extra import platform Co-authored-by: pradeep <pradeep@arrayfire.com>
1 parent 5bead13 commit 70c96ca

File tree

6 files changed

+205
-40
lines changed

6 files changed

+205
-40
lines changed

‎CMakeLists.txt

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
cmake_minimum_required(VERSION 3.11.0)
2+
project(arrayfire-python)
3+
find_package(PythonExtensions REQUIRED)
4+
include(FetchContent)
5+
6+
set(CMAKE_MODULE_PATH_OLD ${CMAKE_MODULE_PATH})
7+
set(CMAKE_MODULE_PATH "")
8+
set(NO_SONAME)
9+
10+
FetchContent_Declare(
11+
arrayfire
12+
GIT_REPOSITORY https://github.com/arrayfire/arrayfire.git
13+
GIT_TAG v3.8
14+
)
15+
FetchContent_MakeAvailable(arrayfire)
16+
set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH_OLD})
17+
18+
set(ignoreWarning "${SKBUILD}")

‎README.md

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -25,30 +25,29 @@ def calc_pi_device(samples):
2525

2626
Choosing a particular backend can be done using `af.set_backend(name)` where name is either "_cuda_", "_opencl_", or "_cpu_". The default device is chosen in the same order of preference.
2727

28-
## Requirements
29-
30-
Currently, this project is tested only on Linux and OSX. You also need to have the ArrayFire C/C++ library installed on your machine. You can get it from the following sources.
28+
## Getting started
29+
ArrayFire can be installed from a variety of sources. [Pre-built wheels](https://repo.arrayfire.com/python/wheels/3.8.0/) are available for a number of systems and toolkits. Wheels for some systems are available on PyPI. Unsupported systems will require separate installation of the ArrayFire C/C++ libraries and only the python wrapper will be installed in that case.
30+
You can get the ArrayFire C/C++ library from the following sources:
3131

3232
- [Download and install binaries](https://arrayfire.com/download)
3333
- [Build and install from source](https://github.com/arrayfire/arrayfire)
3434

35-
Please check the following links for dependencies.
36-
37-
- [Linux dependencies](http://www.arrayfire.com/docs/using_on_linux.htm)
38-
- [OSX dependencies](http://www.arrayfire.com/docs/using_on_osx.htm)
39-
40-
## Getting started
41-
42-
**Install the last stable version:**
4335

36+
**Install the last stable version:**
4437
```
4538
pip install arrayfire
4639
```
4740

48-
**Install the development version:**
41+
**Install a pre-built wheel for a specific CUDA toolkit version:**
42+
```
43+
pip install arrayfire==3.8.0+cu112 -f https://repo.arrayfire.com/python/wheels/3.8.0/
44+
# Replace the +cu112 local version with the desired toolkit
45+
```
46+
47+
**Install the development source distribution:**
4948

5049
```
51-
pip install git+git://github.com/arrayfire/arrayfire-python.git@devel
50+
pip install git+git://github.com/arrayfire/arrayfire-python.git@master
5251
```
5352

5453
**Installing offline:**
@@ -57,10 +56,11 @@ pip install git+git://github.com/arrayfire/arrayfire-python.git@devel
5756
cd path/to/arrayfire-python
5857
python setup.py install
5958
```
59+
Rather than installing and building ArrayFire elsewhere in the system, you can also build directly through python by first setting the `AF_BUILD_LOCAL_LIBS=1` environment variable. Additional setup will be required to build ArrayFire, including satisfying dependencies and further CMake configuration. Details on how to pass additional arguments to the build systems can be found in the [scikit-build documentation.](https://scikit-build.readthedocs.io/en/latest/)
6060

6161
**Post Installation:**
6262

63-
Please follow [these instructions](https://github.com/arrayfire/arrayfire-python/wiki) to ensure the arrayfire-python can find the arrayfire libraries.
63+
If you are not using one of the pre-built wheels, you may need to ensure arrayfire-python can find the installed arrayfire libraries. Please follow [these instructions](https://github.com/arrayfire/arrayfire-python/wiki) to ensure that arrayfire-python can find the arrayfire libraries.
6464

6565
To run arrayfire tests, you can run the following command from command line.
6666

‎arrayfire/library.py

Lines changed: 74 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
import ctypes as ct
1616
import traceback
1717
import os
18+
import sys
1819

1920
c_float_t = ct.c_float
2021
c_double_t = ct.c_double
@@ -492,8 +493,6 @@ class VARIANCE(_Enum):
492493
_VER_MAJOR_PLACEHOLDER = "__VER_MAJOR__"
493494

494495
def _setup():
495-
import platform
496-
497496
platform_name = platform.system()
498497

499498
try:
@@ -527,7 +526,7 @@ def _setup():
527526
ct.windll.kernel32.SetErrorMode(0x0001 | 0x0002)
528527

529528
if AF_SEARCH_PATH is None:
530-
AF_SEARCH_PATH="C:/Program Files/ArrayFire/v" + AF_VER_MAJOR +"/"
529+
AF_SEARCH_PATH="C:/Program Files/ArrayFire/v" + AF_VER_MAJOR +"/"
531530

532531
if CUDA_PATH is not None:
533532
CUDA_FOUND = os.path.isdir(CUDA_PATH + '/bin') and os.path.isdir(CUDA_PATH + '/nvvm/bin/')
@@ -554,7 +553,12 @@ def _setup():
554553
post = '.so.' + _VER_MAJOR_PLACEHOLDER
555554

556555
if AF_SEARCH_PATH is None:
557-
AF_SEARCH_PATH='/opt/arrayfire-' + AF_VER_MAJOR + '/'
556+
if os.path.exists('/opt/arrayfire-' + AF_VER_MAJOR + '/'):
557+
AF_SEARCH_PATH = '/opt/arrayfire-' + AF_VER_MAJOR + '/'
558+
elif os.path.exists('/opt/arrayfire/'):
559+
AF_SEARCH_PATH = '/opt/arrayfire/'
560+
else:
561+
AF_SEARCH_PATH = '/usr/local/'
558562

559563
if CUDA_PATH is None:
560564
CUDA_PATH='/usr/local/cuda/'
@@ -566,21 +570,46 @@ def _setup():
566570
else:
567571
raise OSError(platform_name + ' not supported')
568572

569-
if AF_PATH is None:
570-
os.environ['AF_PATH'] = AF_SEARCH_PATH
571-
572-
return pre, post, AF_SEARCH_PATH, CUDA_FOUND
573+
return pre, post, AF_PATH, AF_SEARCH_PATH, CUDA_FOUND
573574

574575
class _clibrary(object):
575576

577+
def __find_nvrtc_builtins_libname(self, search_path):
578+
filelist = os.listdir(search_path)
579+
for f in filelist:
580+
if 'nvrtc-builtins' in f:
581+
return f
582+
return None
583+
576584
def __libname(self, name, head='af', ver_major=AF_VER_MAJOR):
577585
post = self.__post.replace(_VER_MAJOR_PLACEHOLDER, ver_major)
578586
libname = self.__pre + head + name + post
579-
if os.path.isdir(self.AF_PATH + '/lib64'):
580-
libname_full = self.AF_PATH + '/lib64/' + libname
587+
588+
if self.AF_PATH:
589+
if os.path.isdir(self.AF_PATH + '/lib64'):
590+
path_search = self.AF_PATH + '/lib64/'
591+
else:
592+
path_search = self.AF_PATH + '/lib/'
593+
else:
594+
if os.path.isdir(self.AF_SEARCH_PATH + '/lib64'):
595+
path_search = self.AF_SEARCH_PATH + '/lib64/'
596+
else:
597+
path_search = self.AF_SEARCH_PATH + '/lib/'
598+
599+
if platform.architecture()[0][:2] == '64':
600+
path_site = sys.prefix + '/lib64/'
601+
else:
602+
path_site = sys.prefix + '/lib/'
603+
604+
path_local = self.AF_PYMODULE_PATH
605+
libpaths = [('', libname),
606+
(path_site, libname),
607+
(path_local,libname)]
608+
if self.AF_PATH: #prefer specified AF_PATH if exists
609+
libpaths.append((path_search, libname))
581610
else:
582-
libname_full=self.AF_PATH+'/lib/'+libname
583-
return (libname, libname_full)
611+
libpaths.insert(2, (path_search, libname))
612+
return libpaths
584613

585614
def set_unsafe(self, name):
586615
lib = self.__clibs[name]
@@ -592,13 +621,19 @@ def __init__(self):
592621

593622
more_info_str = "Please look at https://github.com/arrayfire/arrayfire-python/wiki for more information."
594623

595-
pre, post, AF_PATH, CUDA_FOUND = _setup()
624+
pre, post, AF_PATH, AF_SEARCH_PATH, CUDA_FOUND = _setup()
596625

597626
self.__pre = pre
598627
self.__post = post
599628
self.AF_PATH = AF_PATH
629+
self.AF_SEARCH_PATH = AF_SEARCH_PATH
600630
self.CUDA_FOUND = CUDA_FOUND
601631

632+
# prefer locally packaged arrayfire libraries if they exist
633+
af_module = __import__(__name__)
634+
self.AF_PYMODULE_PATH = af_module.__path__[0] + '/' if af_module.__path__ else None
635+
636+
602637
self.__name = None
603638

604639
self.__clibs = {'cuda' : None,
@@ -618,7 +653,7 @@ def __init__(self):
618653
'opencl' : 4}
619654

620655
# Try to pre-load forge library if it exists
621-
libnames = self.__libname('forge', head='', ver_major=FORGE_VER_MAJOR)
656+
libnames = reversed(self.__libname('forge', head='', ver_major=FORGE_VER_MAJOR))
622657

623658
try:
624659
VERBOSE_LOADS = os.environ['AF_VERBOSE_LOADS'] == '1'
@@ -628,14 +663,15 @@ def __init__(self):
628663

629664
for libname in libnames:
630665
try:
631-
ct.cdll.LoadLibrary(libname)
666+
full_libname = libname[0] + libname[1]
667+
ct.cdll.LoadLibrary(full_libname)
632668
if VERBOSE_LOADS:
633-
print('Loaded ' + libname)
669+
print('Loaded ' + full_libname)
634670
break
635671
except OSError:
636672
if VERBOSE_LOADS:
637673
traceback.print_exc()
638-
print('Unable to load ' + libname)
674+
print('Unable to load ' + full_libname)
639675
pass
640676

641677
c_dim4 = c_dim_t*4
@@ -644,24 +680,39 @@ def __init__(self):
644680

645681
# Iterate in reverse order of preference
646682
for name in ('cpu', 'opencl', 'cuda', ''):
647-
libnames = self.__libname(name)
683+
libnames = reversed(self.__libname(name))
648684
for libname in libnames:
649685
try:
650-
ct.cdll.LoadLibrary(libname)
686+
full_libname = libname[0] + libname[1]
687+
688+
ct.cdll.LoadLibrary(full_libname)
651689
__name = 'unified' if name == '' else name
652-
clib = ct.CDLL(libname)
690+
clib = ct.CDLL(full_libname)
653691
self.__clibs[__name] = clib
654692
err = clib.af_randu(c_pointer(out), 4, c_pointer(dims), Dtype.f32.value)
655693
if (err == ERR.NONE.value):
656694
self.__name = __name
657695
clib.af_release_array(out)
658696
if VERBOSE_LOADS:
659-
print('Loaded ' + libname)
697+
print('Loaded ' + full_libname)
698+
699+
# load nvrtc-builtins library if using cuda
700+
if name == 'cuda':
701+
nvrtc_name = self.__find_nvrtc_builtins_libname(libname[0])
702+
if nvrtc_name:
703+
ct.cdll.LoadLibrary(libname[0] + nvrtc_name)
704+
705+
if VERBOSE_LOADS:
706+
print('Loaded ' + libname[0] + nvrtc_name)
707+
else:
708+
if VERBOSE_LOADS:
709+
print('Could not find local nvrtc-builtins libarary')
710+
660711
break;
661712
except OSError:
662713
if VERBOSE_LOADS:
663714
traceback.print_exc()
664-
print('Unable to load ' + libname)
715+
print('Unable to load ' + full_libname)
665716
pass
666717

667718
if (self.__name is None):
@@ -689,6 +740,7 @@ def parse(self, res):
689740
lst.append(key)
690741
return tuple(lst)
691742

743+
692744
backend = _clibrary()
693745

694746
def set_backend(name, unsafe=False):

‎pyproject.toml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
[build-system]
2+
requires = ["setuptools", "wheel", "scikit-build", "cmake", "ninja"]

‎setup.cfg

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,10 @@
11
[metadata]
22
name = arrayfire
3-
version = 3.7.20200213
3+
version = 3.8.0
44
description = Python bindings for ArrayFire
55
licence = BSD
66
long_description = file: README.md
7+
long_description_content_type = text/markdown
78
maintainer = ArrayFire
89
maintainer_email = technical@arrayfire.com
910
url = http://arrayfire.com
@@ -15,6 +16,8 @@ classifiers =
1516

1617
[options]
1718
packages = find:
19+
install_requires=
20+
scikit-build
1821

1922
[options.packages.find]
2023
include = arrayfire

‎setup.py

Lines changed: 93 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,8 +9,98 @@
99
# http://arrayfire.com/licenses/BSD-3-Clause
1010
########################################################
1111

12-
# TODO: Look for af libraries during setup
12+
import os
13+
import re
1314

14-
from setuptools import setup
15+
# package can be distributed with arrayfire binaries or
16+
# just with python wrapper files, the AF_BUILD_LOCAL
17+
# environment var determines whether to build the arrayfire
18+
# binaries locally rather than searching in a system install
19+
20+
AF_BUILD_LOCAL_LIBS = os.environ.get('AF_BUILD_LOCAL_LIBS')
21+
print(f'AF_BUILD_LOCAL_LIBS={AF_BUILD_LOCAL_LIBS}')
22+
if AF_BUILD_LOCAL_LIBS:
23+
print('Proceeding to build ArrayFire libraries')
24+
else:
25+
print('Skipping binaries installation, only python files will be installed')
26+
27+
AF_BUILD_CPU = os.environ.get('AF_BUILD_CPU')
28+
AF_BUILD_CPU = 1 if AF_BUILD_CPU is None else int(AF_BUILD_CPU)
29+
AF_BUILD_CPU_CMAKE_STR = '-DAF_BUILD_CPU:BOOL=ON' if (AF_BUILD_CPU == 1) else '-DAF_BUILD_CPU:BOOL=OFF'
30+
31+
AF_BUILD_CUDA = os.environ.get('AF_BUILD_CUDA')
32+
AF_BUILD_CUDA = 1 if AF_BUILD_CUDA is None else int(AF_BUILD_CUDA)
33+
AF_BUILD_CUDA_CMAKE_STR = '-DAF_BUILD_CUDA:BOOL=ON' if (AF_BUILD_CUDA == 1) else '-DAF_BUILD_CUDA:BOOL=OFF'
34+
35+
AF_BUILD_OPENCL = os.environ.get('AF_BUILD_OPENCL')
36+
AF_BUILD_OPENCL = 1 if AF_BUILD_OPENCL is None else int(AF_BUILD_OPENCL)
37+
AF_BUILD_OPENCL_CMAKE_STR = '-DAF_BUILD_OPENCL:BOOL=ON' if (AF_BUILD_OPENCL == 1) else '-DAF_BUILD_OPENCL:BOOL=OFF'
38+
39+
AF_BUILD_UNIFIED = os.environ.get('AF_BUILD_UNIFIED')
40+
AF_BUILD_UNIFIED = 1 if AF_BUILD_UNIFIED is None else int(AF_BUILD_UNIFIED)
41+
AF_BUILD_UNIFIED_CMAKE_STR = '-DAF_BUILD_UNIFIED:BOOL=ON' if (AF_BUILD_UNIFIED == 1) else '-DAF_BUILD_UNIFIED:BOOL=OFF'
42+
43+
if AF_BUILD_LOCAL_LIBS:
44+
# invoke cmake and build arrayfire libraries to install locally in package
45+
from skbuild import setup
46+
47+
def filter_af_files(cmake_manifest):
48+
cmake_manifest = list(filter(lambda name: not (name.endswith('.h')
49+
or name.endswith('.cpp')
50+
or name.endswith('.hpp')
51+
or name.endswith('.cmake')
52+
or name.endswith('jpg')
53+
or name.endswith('png')
54+
or name.endswith('libaf.so') #avoids duplicates due to symlinks
55+
or re.match('.*libaf\.so\.3\..*', name) is not None
56+
or name.endswith('libafcpu.so')
57+
or re.match('.*libafcpu\.so\.3\..*', name) is not None
58+
or name.endswith('libafcuda.so')
59+
or re.match('.*libafcuda\.so\.3\..*', name) is not None
60+
or name.endswith('libafopencl.so')
61+
or re.match('.*libafopencl\.so\.3\..*', name) is not None
62+
or name.endswith('libforge.so')
63+
or re.match('.*libforge\.so\.1\..*', name) is not None
64+
or 'examples' in name), cmake_manifest))
65+
return cmake_manifest
66+
67+
print('Building CMAKE with following configurable variables: ')
68+
print(AF_BUILD_CPU_CMAKE_STR)
69+
print(AF_BUILD_CUDA_CMAKE_STR)
70+
print(AF_BUILD_OPENCL_CMAKE_STR)
71+
print(AF_BUILD_UNIFIED_CMAKE_STR)
72+
73+
74+
setup(
75+
packages=['arrayfire'],
76+
cmake_install_dir='',
77+
cmake_process_manifest_hook=filter_af_files,
78+
include_package_data=False,
79+
cmake_args=[AF_BUILD_CPU_CMAKE_STR,
80+
AF_BUILD_CUDA_CMAKE_STR,
81+
AF_BUILD_OPENCL_CMAKE_STR,
82+
AF_BUILD_UNIFIED_CMAKE_STR,
83+
# todo: pass additional args from environ
84+
'-DCMAKE_BUILD_TYPE:STRING="RelWithDebInfo"',
85+
'-DFG_USE_STATIC_CPPFLAGS:BOOL=OFF',
86+
'-DFG_WITH_FREEIMAGE:BOOL=OFF',
87+
'-DCUDA_architecture_build_targets:STRING=All',
88+
'-DAF_BUILD_DOCS:BOOL=OFF',
89+
'-DAF_BUILD_EXAMPLES:BOOL=OFF',
90+
'-DAF_INSTALL_STANDALONE:BOOL=ON',
91+
'-DAF_WITH_IMAGEIO:BOOL=ON',
92+
'-DAF_WITH_LOGGING:BOOL=ON',
93+
'-DBUILD_TESTING:BOOL=OFF',
94+
'-DAF_BUILD_FORGE:BOOL=ON',
95+
'-DAF_INSTALL_LIB_DIR:STRING=arrayfire',
96+
'-DAF_INSTALL_BIN_DIR:STRING=arrayfire',
97+
'-DFG_INSTALL_LIB_DIR:STRING=arrayfire',
98+
'-DAF_WITH_STATIC_MKL=ON',
99+
]
100+
)
101+
102+
else:
103+
# ignores local arrayfire libraries, will search system instead
104+
from setuptools import setup
105+
setup()
15106

16-
setup()

0 commit comments

Comments
(0)

AltStyle によって変換されたページ (->オリジナル) /