Fragment - Collaborative Spectral/Granular/Subtractive/PM Synthesizer

All your LV2 and LADSPA goodness and more.

Moderators: MattKingUSA, khz

Onirom
Established Member
Posts: 16
Joined: Tue Jan 24, 2017 11:17 pm

Fragment - Collaborative Spectral/Granular/Subtractive/PM Synthesizer

Post by Onirom »

Hello everyone,

i want to present you an experiment, Fragment, a collaborative and free spectral musical instrument.

Here is a YouTube play-list of all demo videos released so far :

https://www.youtube.com/playlist?list=P ... KbvCzkjgeI

This is a synthesizer which associate visuals/audio through direct manipulation of the spectrum, the visuals are generated by a GLSL script (the GPU is producing the visuals) which is shared between users and per sessions, the visuals represent a kind of possibilities space from which you choose what to hear by "slicing it", spectrum slices are fed to a pure additive synthesis engine in real-time, i believe that some weird sounds can be made easily with this synthesizer, complex sounds can also be made but it may require higher knowledges.

Visuals/sounds can be manipulated by tweaking the script variables directly or through MIDI enabled controls widgets (this is a Chrome and Opera only feature because FireFox does not implement it right now), it is also possible to manipulate the spectrum with your camera or images, that way it is possible to draw a score on a piece of paper or anything else and play it back with your camera or by adding the image and messing with GLSL scripts, anything can be made.

This is mostly web-based but there is a linux app. which provide crackles free performances (due to web-based performance issues for now), i am also in the process to release a standalone all-in-one executable for Linux and other platforms which will allow crackles free performances directly.

This synth. require some knowledges of GLSL in order to use it, you can follow the example comments for some hints and the help dialog for some helps.

If you have any questions, i will be glad to answer them here.

You can try it at : https://www.fsynth.com
Last edited by Onirom on Sun Dec 31, 2017 5:17 pm, edited 4 times in total.
User avatar
davephillips
Established Member
Posts: 592
Joined: Sat Aug 15, 2015 1:05 pm
Has thanked: 35 times
Been thanked: 23 times

Re: Fragment - The Collaborative Spectral Synthesizer

Post by davephillips »

Greetings,

This looks very cool, I'm looking forward to checking it out. :)

Best,

dp
User avatar
Michael Willis
Established Member
Posts: 1451
Joined: Mon Oct 03, 2016 3:27 pm
Location: Rocky Mountains, North America
Has thanked: 69 times
Been thanked: 164 times
Contact:

Re: Fragment - The Collaborative Spectral Synthesizer

Post by Michael Willis »

Very neat!
stanlea
Established Member
Posts: 700
Joined: Wed Apr 25, 2012 9:49 pm
Has thanked: 41 times
Been thanked: 23 times

Re: Fragment - The Collaborative Spectral Synthesizer

Post by stanlea »

Live coding is always nice. The demo seems promising. Thanks.

edit : why wouldn't you make a presentation of it at next Linux Audio Conference ?

https://lac2017.univ-st-etienne.fr/fr/36-2/
Onirom
Established Member
Posts: 16
Joined: Tue Jan 24, 2017 11:17 pm

Re: Fragment - The Collaborative Spectral Synthesizer

Post by Onirom »

Thank you, i will consider this but i am pretty busy at the moment :)

Btw, Fragment was updated today with a new additive synthesis engine which (almost) annihilate the unbearable audio crackles when the audio is played on the browser side, i also added a real-time logarithmic spectrum view of the output.

Consider using Chromium/Chrome for best performances, it is much better now in FireFox as well though, there is some work left to polish that new additive synthesis engine and make it faster. The goal right now is to annihilate the use of an external program to play the audio perfectly. :D
jonetsu
Established Member
Posts: 2036
Joined: Sat Jun 11, 2016 12:05 am
Has thanked: 10 times
Been thanked: 22 times

Re: Fragment - The Collaborative Spectral Synthesizer

Post by jonetsu »

The way the colors and shapes seems to move organically with the music is very interesting. There might be a lot more to see with this technology, so to speak.

Salutations.
jonetsu
Established Member
Posts: 2036
Joined: Sat Jun 11, 2016 12:05 am
Has thanked: 10 times
Been thanked: 22 times

Re: Fragment - The Collaborative Spectral Synthesizer

Post by jonetsu »

Not perhaps exactly the same application, but still in live coding, there is also Praxis Live.

https://www.youtube.com/watch?v=DGKrCdGIOx4

Debian installer available.

http://www.praxislive.org/

There is actually an annual conference on live coding. The 2015/2016 proceedings:

http://iclc.livecodenetwork.org/2015/papers.html

http://iclc.livecodenetwork.org/2016/papers.html
Onirom
Established Member
Posts: 16
Joined: Tue Jan 24, 2017 11:17 pm

Re: Fragment - The Collaborative Spectral Synthesizer

Post by Onirom »

Thank you for the link, even if it is not really the same thing, i did not know Praxis Live and it seem pretty great! :)

After some experimentation with the Raspberry Pi, i made a video of a live jam which show how Fragment can be used in a DAW such as Renoise (the audio run on a Raspberry Pi) :

https://www.youtube.com/watch?v=7x9S0sCz2GU
Onirom
Established Member
Posts: 16
Joined: Tue Jan 24, 2017 11:17 pm

Re: Fragment - The Collaborative Spectral Synthesizer

Post by Onirom »

Here is some new videos of some jam sessions which show how to play with Fragment via a MIDI keyboard (a recent feature) :

https://www.youtube.com/watch?v=taR6aAJt3ko

https://www.youtube.com/watch?v=IzfofbrU8XE

https://www.youtube.com/watch?v=nIIkDxdp5hk

Here is a quick track from a jam session (all synth. sounds come from Fragment, drums and fx were added with Renoise) :

https://soundcloud.com/fsynth/jam-session

While i was jamming with it today, found out that there is some limitations, notably the fact that you cannot have multiple instances of the synthesizer (because of performances among other things), it limit you to have one kind of sound at a time mostly thus my next focus will be to implement output settings for specific spectrum slices (this allow routing to different tracks within a sequencer) and allow to determinate controllers/MIDI channel from the shader code which will enable an unlimited number of sounds to be produced by a single instance from multiple keyboards/sequencers, my goal is to use it as a virtual instrument in Renoise but i presume that it will be very handy to have all those features to use Fragment with any other hardware or software sequencers or controllers!

The standalone version (much better performances) and the complete software documentation with examples and tutorials is also in progress.
Onirom
Established Member
Posts: 16
Joined: Tue Jan 24, 2017 11:17 pm

Re: Fragment - The Collaborative Spectral Synthesizer

Post by Onirom »

Hello,

Fragment had a massive update recently, the limitations are no more and Fragment is now a fully capable multitimbral synthesizer with multi-output audio per slices, this work best with Linux, Jack and the Fragment Audio Server, the update in details:
  • multitimbral abilities through accessible MIDI channel values (kinda limited by MIDI 16 channels spec. right now though)
  • live-coding features (GLSL errors don't stop audio/visuals anymore)
  • MIDI devices dialog which can be used to enable/disable MIDI devices
  • hot plugging of MIDI devices is now supported
  • extended polyphony, only limited by the GPU (polyphony capability is automatically detected from the GPU)
  • configurable output channel per slices, this allow Fragment to work like any modern audio plugins and most importantly like a real synthesizer would work
  • monaural mode, this is a very important feature because it allow to produce synchronized visuals to work alongside the audio that is generated, in monaural mode, the complete RGB output components are available to do visuals stuff (and the alpha component is used for the audio synthesis)
  • fullscreen code editor with transparent background by pressing F11 when the editor is focused
  • new settings to show actual polyphony and osc. count
  • new settings to hide/show the ugly slices bar or hide the slices completely
  • new user interface guide (available on the help dialog)
  • new website
  • documentation (still WIP for GLSL section but good enough to be released)
Here is a video : https://www.youtube.com/watch?v=VwQdMRVxkpY

The video show most recent features and a simple use of webcam input.
The video also show per slices increment, a new feature which is not official yet.

And most importantly, here is link to a draft of the documentation:

https://www.fsynth.com/documentation

Any suggestions/ameliorations/fix/contributions are welcome!
Onirom
Established Member
Posts: 16
Joined: Tue Jan 24, 2017 11:17 pm

Re: Fragment - The Collaborative Spectral Synthesizer

Post by Onirom »

Hello,

Fragment has been updated numerous times since then, here are some of the major features :
  • Canvas inputs make it possible to draw on the canvas with brushes, textures are then updated in real-time, Fragment is bundled with 970 high-quality brushes, there is many painting options such as randomize, axis locking, brush scaling, brush opacity, brush spacing, brush angle and compositing
  • An import dialog make it possible to import audio files (which are then converted to spectrogram, Fragment is able to do crude re-synthesis) and images in batch by drag & drop, it also provide numerous settings to fine tune the analysis of audio files
  • Spectral recording dialog, a continuous frame by frame pixels recording of what is played by Fragment with simple compositing, the recorded image can then be exported as a PNG or imported back as image to be used by the synthesizer
  • Feedback support, this make the previous frame available as a texture to do advanced effects such as multi-tap delay, reverb and advanced sonic textures
  • Fragment inputs can be re-ordered on the fly (similarly to turntables)
  • Band-limited noise was added to the additive synthesis engine of the server program
  • Slices has a "per frame increment" settings which allow them to move left or right automatically
  • GLSL 3 support
  • GLSL outline dialog, show a list of functions and variables of the fragment shader with quick jump
  • Documentation is relatively complete, just missing more examples
The first post has been updated with a direct link to the YouTube play-list.
Onirom
Established Member
Posts: 16
Joined: Tue Jan 24, 2017 11:17 pm

Re: Fragment - The Collaborative Spectral Synthesizer

Post by Onirom »

Hello,

Fragment has been updated with some major features :
  • 32-bits float image data (WebGL 2 compatible browser only), this allow better sound quality by allowing higher range of amplitude values
  • support for full RGB visuals alongside stereo audio
  • pKey pre-defined uniform for portamento effects
  • many more small fixes and minor features such as normalization of imported audio images
There is also a graphical launcher for the audio server, although it is not distributed on the official site right now :

https://github.com/grz0zrg/fas_launcher

Also, a way has been found to remove the refresh-rate slices capture by running the browser without VSYNC, there is a video below which demonstrate 150fps slices capture, this allow better events granularity.
Onirom
Established Member
Posts: 16
Joined: Tue Jan 24, 2017 11:17 pm

Re: Fragment - Collaborative Spectral/Granular Synthesizer

Post by Onirom »

Hello,

Fragment is now one year old and to celebrate this, a massive 1.0 anniversary update was published along with new documentation, tools, homepage and videos.

To make it easier for newcomer to try it out with maximum performances, a .deb package is available on the homepage, this install the audio server and a launcher for your system (Ubuntu, Debian etc.).

Here are some new videos :

Granular + Additive synthesis & visuals ambient soundscape
live additive synthesis with Open Stage Control/OSC & videos compositing

Here is the client changelog :
  • complete granular synthesis support, asynchronous and synchronous (audio server only)
  • videos import with image conversion of the audio part and playback/looping settings
  • note lifetime : The ability to fully handle the release part of notes, a part which was a bit cumbersome before
  • OSC IN : GLSL uniforms can be controlled through OSC, videos can be controlled through OSC
  • OSC OUT : The notes data can be sent through OSC bundles (a SuperCollider additive/spectral synth was made as a demonstration of the possibilities)
  • distributed sound synthesis multi-machines/multicore with fas_relay tool and audio server instances
  • new easy to use launcher tool for the audio server/app. this is distributed on the homepage with a nice .deb packages for Linux distributions
  • controllers were removed due to OSC IN support (i recommend using the Open Stage Control software)
  • new documentation made with mkdocs
  • new homepage
  • many optimizations
  • phase information kept in alpha channel of imported textures
  • direct support to link the audio server and the application through the session URL (append ?fas=1 URL arg)
  • more analysis window (kaiser, nutall, flattop) along with alpha parameter for kaiser, gauss and blackman
  • the audio server CPU stream load is now displayed in Fragment
  • facelift of the help dialog with code snippets (ADSR etc), about tab and click-to-copy clipboard feature
  • the demo code was updated
  • fix decimals issue with input widget
  • fix inputs save/restore
  • update ShaderToy conversion feature
  • add fmov pre-defined uniform (video input playback time access from the fragment shader)
  • improved notification system/GLSL compilation report
The Fragment Audio Server also get some new specific features and major fixes :
  • improved sound quality through configurable linear interpolation factor, this was static before and contributed to muddy sound quality
  • enhanced stability, all known stability issues were fixed
  • no more pop sound on first connection with additive synthesis
  • hot-reload of samples
  • grains pitch can be detected from patterns in the filename (MIDI note or user-defined frequency)
  • improved bandwidth- enhanced oscillators
  • many other minor improvements
All planned features were implemented, the future is made of more synthesis methods, improved collaborative features along with improved ease of use and optimizations. :)
Onirom
Established Member
Posts: 16
Joined: Tue Jan 24, 2017 11:17 pm

Re: Fragment - Collaborative Spectral/Granular Synthesizer

Post by Onirom »

Hello,

Fragment v1.0.1/v1.0.2 has been released.

The two new major features are MIDI Multidimensional Polyphonic Expression (MPE) support and much more improved audio engine (no more crackles!)

Here is the client changelog :
  • MIDI MPE support, Fragment expose aftertouch, CC74 and pitch bend data along with key release velocity, the MIDI code was also improved/optimized
  • In-app quickstart guide & MOTD, sample program is less cluttered with comments
  • more snippets added such as polar/cartesian conversion, pi, simple filter etc.
  • improved look & feel of the windows launcher
  • action snippets : "remove comments" and "remove spaces"
  • added a support tab
  • fix issue with images download on Chrome and related browser
  • fix issue with batch import
  • fix issue with samples reload
  • fix issue with wavetable method on Firefox
  • fix issue with channels settings saving
  • homepage update
  • documentation update
Here is the audio server changelog :
  • improved sound engine, no more crackles should be heard
  • improved granular synthesis, many issues related to grains playback were resolved resulting in near perfect playback
  • improved samples loading/playback, a fade in/out is applied on all loaded samples and samplerate conversion is applied
  • many small optimization and code cleanup/improvement were also done
Two new videos were also published (note : recorded before both update thus imperfect grains playback/crackles :D ) :

Various additive/granular sound design, this is a melting pot video of sounds produced since the development of Fragment from newest to oldest :

https://www.youtube.com/watch?v=2t72Me_hmUM

Getting started tutorial, additive + granular ambient soundscape workflow : This video is not edited and serve as a guide at the beginning with explanation of how to launch the audio server and produce some sounds, this is a full jam based on the same setup than the 1.0 launch video.

https://www.youtube.com/watch?v=hrHlO2gNfHw

Now that the audio engine is quite smooth, one of the 'last' thing which i need to implement on the server side is an offline rendering feature which will allow to record the audio smoothly to disk, on the collaborative aspect, a new tool will be released in the coming month which will allow to share sound design or even compositions/jam made with Fragment (or any other image-synth)
Onirom
Established Member
Posts: 16
Joined: Tue Jan 24, 2017 11:17 pm

Re: Fragment - Collaborative Spectral/Granular/Subtractive/PM Synthesizer

Post by Onirom »

Hello,

Fragment was updated today with added support for subtractive (Moog filter)/PM synthesis, improved granular synthesis and analysis and many other changes.
  • subtractive and PM synthesis can now be selected in the FAS settings dialog
  • real-time recording can now be exported to .WAV from the recording dialog (with additive synthesis only)
  • mipmap input filter is now available
  • added interpolation between bins to increase sonogram quality of imported sounds
  • move FAS settings dialog button to a proper button in the main toolbar
  • much better looking quickstart guide and revamped help dialog
  • enhanced fullscreen mode editor
  • fix some issues when FAS is enabled
  • fix background opacity when the editor is in fullscreen mode
  • fix issue with record canvas
  • updated documentation
The audio server was also updated for all platforms :
  • add phase modulation synthesis
  • add subtractive synthesis with Moog type filter and 3 waveforms (triangle / sawtooth / square)
  • granular synthesis : improved & optimized on many levels, grains samples can now be changed in real-time without crackles
  • granular synthesis : grains are now resampled with linear interpolation
  • fix granular synthesis issues with envelopes and negative grains speed
  • fix issue with stereo samples loading
  • Valgrind was used to hunt down all memory leaks
FAS Relay (distributed synthesis) was updated with minor fixes.

Two new videos were also published demonstrating the higher quality sound engine and new synthesis methods :

https://youtu.be/nB_g0mIc4_w

Granular/Additive soundscapes & pixels-based delay : a live additive + granular ambient soundscapes session which demonstrate pixels-based delay along with some kind of "vector based synthesis" by using a XY widget in Open Stage Control (through OSC) with the 4 channels volume mapped onto the widget axis resulting in cross-fading of the 4 channels.

https://youtu.be/oN3CS_nkwD8

Granular/Additive/Subtractive/PM real-time chaos : a live PM/Subtractive/Additive/Granular ambient chaotic & glitchy session which demonstrate the two new type of sound synthesis : subtractive synthesis and phase modulation, both with Moog type filter. A camera glitch visual effect is also applied and modified in real-time along the produced sound to produce glitchy visuals. The video show 4 timbral parts from left to right : phase modulation, subtractive, additive, granular

https://youtu.be/WvV9TkUfByE Part 1

https://www.youtube.com/watch?v=GrkbXVZW870 Part 2

Granular/Additive visuals reactive audio : a live Subtractive/Additive/Granular ambient session which demonstrate visuals reactive audio, a MIDI sequence is played and recorded as an image inside Fragment, the produced image is then drawn onto the canvas two times at different offset, a compositing function is applied between the images and the video, this mostly result in audio sounding "bigger" when the brightness is high such as on the antenna/blue sky part, this effect can be heard the most on the antenna part @4:06 different compositing functions are used along the video, the part 2 video has a more exprimental approach by using the video pixels directly and feeding it to the granular/additive synthesizer.

Best wishes to all for 2018!
Post Reply